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PREFACE 

I. THE CONTEXT OF THE ISSUE 

Before beginning our analysis of the appropriate balance be- 
tween "Field-Initiated and Agency-Directed" R&D, it will be impor- 
tant to frame the issue in its proper context. This can be intro- 
duced by a brief discussion of some central que.i.tions for NIE today. 

What are NIE's responsibilities towards U.S. education in 
general and educational R&D in particular? How can the Agency 
utilize a very limited budget for the best short and long term 
effect, and how might it justify a proper increase in this budget? 
These are central policy questions, as are such related issues as: 
the number and types of personnel required within the Agency; the 
proper place of an in-house Research effort (if any); how NIE 
should relate to the "Field"; the current priorities for NIE in 
the "Field"; and NIE's relations with other Agenc.es that may be 
playing roles in either educational R&D or in educational practice. 

Any attempt to deal with these issues must begin with a rec- 
ognitidii of their inter-connectedness. For all its looseness, its 
inadequacies, there is an educational R&D system to be dealt with 
and NIE is a critical element of that system. What is done in one 
area or in relation to one issue will likely affect other areas 
and have impact on other issues. For example, the build-up of 
development efforts in the past has had implications for the state 
of Applied Research and the needs and opportunities for Dissemina- 
tion now. What the Agency adopts as its mission will determine its 
budgetary priorities and in turn how it should and could relate to 
the Field. 

The recognition of the current state of educational R&D (in- 
cluding the total innovation process) is as critical as the rec- 
ognition of its systemic character. It ^s loose; gaps are 
characteristic; and inadequacies are all too common. In short, it 
is a very "immature" and weak R&D, system. What then are the im- 
plications of these two characteristics of being a system and being 
immature and weak? 
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If education is to be served by a quality R&D system, two major 
requirements will need to be satisfied. These involve (a) system build- 
ing, maintenance and protection and (b) system orchestration • 

Only futility and frustration can come from policies that ignore the 
state of the educational R&D system; policies which implicitly assume: 
viable Research/Development/Dissemination and other institutions which 
are reasonably well linked to each ether and to practice; policies which 
assume that Users are able to adopt quality R&D outputs, able to properly 
generate and implement their own significant innovations and able to iden- 
tify and feed forward their real needs to Developers, etc.; policies which 
assume that the provision of funds to procure R&D outputs and prograias are 
the primary requirement for success. We suggest, in contrast, that system 
building, institution building and rebuilding, and personnel development 
are top current priorities for educational R&D. Further, it is not enough 
merely to build. A fragile, politically exposed and weak system must be 
maintained and protected. 

A mature R&D system orchestrates itself. Relationships are well de- 
veloped. Participants know what to seek and to deliver, from and to where, 
and x^at to expect and trust. An immature system needs help to grow, to 
learn how to achieve such a self organizing state. 

These are the needs. NIE may not be the largest governm^intal contrib- 
utor to the educational R&D establishment, but it is the lead, the core di 
cretionary cgency. NIE must take responsibility for the system building 
and orchestrating. No other body can or will. NIE could walk away or be 
prevented, politically, from assuming this mission; but the need would 
remain, as well as the ensuing frustration. In our anralysis we have taken 
the assumption of this mission as a given, along with our premise that 
there is an educational R&D system, albeit immature and weak. 

This sets th-;^ scene for our analysis. If we are to understand the 
factors that determine the what, when, how much and how of the "Field- 
Initiated vs Agency^)irected" R&D issue, it must be in terms of our under- 
standing o:;." the total educational R&D system as it now is; as we might 
wish it to become; and as It varies across the differing elements of the 
system (e.g. as between Research and Development). To do this we need an 
appreciation of what is fundemental and gener*ic in R&D and what is charac- 
teristic \of the present educational R&D context. 



Such understanding will need development and presentation if 
we are. to be able to deal with the issue at hand. It may, inci- 
dently (because of the aforementioned inter-connectedness of 
system issues) also provide some insight to the type of questions 
raised at the start of this preface. Some very brief thoughts on 
those questions may be a fitting introduction and entree into our 
analysis, 

II. SOME RELATED ISSUES 

The limited size of NIE's budget in relation to the needs, and 
even in comparison to what is being spent in total on educational 
R&D, is well recognized in the Agency and in the National Council 
on Educational Research. What are some insights that may be help- 
ful in developing parameters and guidelines for budgetary planning 
that may be derived from the analysis that we present below? 

This is clearly a large policy issue cind we can only hope to 
suggest some useful perspectives in these few comments. 

We would need to begin with an evaluation of the capacity of 
the R&D system, overall and across its parts (or functions as we 
will be terming them— Basic Research, Problem-Focused Research, 
Development, Dissemination, etc.)* What is there now (in terms of 
capacity, product inventories, etc.)? What can be delivered? What 
is needed (capacity^ outputs) now and over the longer term? In our 
analysis we will point out the needs and requirements of the diff- 
erent functions — and how these may differ over time. 

Specifically, in our analysis we will note that because of 
funding policies in relation to the Development function, there are 
some quality Development organizations and an inventory of Develop- 
ment products. What has been missing have been the quality control 
function, the Developer/User linkage and the User/Product matching 
(and tailoring) functions. We will further note in our analysis 
that the Problem-Focused Research and (until recently) the Dis- 
semination functions have not been well developed and are weak. 
We will note that overall the educational R&D system is relatively 
imccature • 
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Given these emiditions and the consequent nfeed- for system- build- ' 
ing, how can NIE best allocate its financial resources? To answer this 
question requires consideration of several factors. For exampla: 

1. There are inherent differences in funding requirementsJhatween 
functions. For example, both institution building and project 
costs in Basic Research tend to be lower than for Problem Focussd 
research which in turn tends to require less funding than 
Development, i.e. the incremental impact of funding additions 
and reductions is variable. 

2. The time required to build a system will vary across functions. 
For example, the time required in Basic Research would be quite 
significantly greater than for Development. Relatedly, so are 
the time horizon needs and impact of funding changes. Skills 
take longer to build and the disruption caused by funding 
variations have much longer consequences. 

3. As Sieber (1975) has noted, in the system building process there 
needs to be a balance in the total system among functions, per- 
sonnel, institutions, decisionmakers, and supply and demand. 

4. NIE is neither the only nor the largest source of tuuding for 
educational R&D. Thus, on the one hand, NIE could attempt to 
increase its leverage through coordinati on and orchestration 
with other agencies to achieve multiplier and syix4rgi«Lic effects. 
On the other hand, NIE budget allocations might f^cus on gap 
filling in areas not funded by other agencies. Thus, it is 
possible for the NIE budget to reflect cross-agency opportun- 
ities as well as NIE priorities and levels of effort. 

5. Consideration must be given to the minimal (floo^) level of 
funding needed to maintain quality and stabilitywithin a func- 
tion, and to the maximum (ceiling) level of funding that can 

be absorbed by a function (given its current state) . As we will 
note. Research system building rates and the ability to spend 
funds productively is limited by the extent of the existing 
centers of exellence. Relatedly, funding for system building 
may nead to be concentratec. rather than scattered around. 
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To illustrate how these budgetary parameters might be applied, 
let us consider what a "balanced'* funding process for Research, 
Development and Dissemination-: might be, given the existing state of 
these system functions. 

As we noted, a relatively large amount of funding has been pro- 
vided iti the past for Development, while Problem-Focused Research 
and (until recently) Dissemination has been relatively less developed. 

Thus, a "balanced" budget plan might be: 

1. Since there are now a sizeable number of Development 
products available, reduce current Development fund- 
ing--to that minimal level necessary to maintain the 
existing high quality centers. 

2. Since Dissemination has b^^n so 'gcrently fragment- 
ed, direct significant funding to Dissemination — but 
not so much as to build a system that would overwhelm 
Users. At this time, quality control,, sorting and 
technical service would likely need to be a part of 
the function. 

3. Since large scale Problem-Focused Research has been neg- 
lected, provide major funding here 'for system building — 
but not at a level greater than the capacity of the 

- — function to absorb productively., 

4. Provide moderate funding for Basic Research, for long- 
term system-building purposes. 

4 A funding strategy such as the above would, of c urse, have to 
consider existing funding realities, political conditions and the 
particular current needs of Users. 

Additional considerations will be important for such a " system 
building" budget and policy orientation ^ For examples 

1. There must be funding stability over time . System 
building is a sustained rather than an "in-and-out" 
process. A three to five year period would be minimal 
for any kind of system building— and would be com- 
pletely inadequate in the Research function. For total 
system building, a much longer time frame is required. 
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1. System building is different from procx-xrement of a product, 
and this fact has significant implications for funding, 
policies. For the procurement of a product, open competi- 
tive bidding is of ten a systemi'cally" v the 
product (not an institution) is the concern. In system building 
ing, the reverse holds true — the institutional (and persom;el) 
base is the primary concern — not a product. 

3. From the political point of view it may be vital to attempt to 
educate the relevant communities as to the state of the edu- 
cational R&D system and to the fact that the next few years 
have to be seen as a period of lon g term capital investmerxt-- * 
if we are not to be burdened in the future with the errors of 
the past as we seem to be today. Perhaps this is the only 
meaningful justification that can be used for added Agency 
funding that will not return to plague educational R&D in the 
- near future. 

These last points warrant further consideration. As we will note in 
the analysis, system building requires continuity and concentration . Di- 
rection and orchestration must be provided from some system-wide agency 
such as NIE. Thus, Agency efforts cannot be scattered and non-directive 
and still be effective. This, in turn, implies a tighter degree of selec- 
tivity and control by the Agency than would be possible under standard RFP 
and competitive bidding procedures. Thus, there is a dilemma — there are 
legal and political constraints involved in a "sole^ source'' approach (which 
would be a relevant mode of funding for system building purposes) . 

Thus, as our analysis will indicate, it becomes important for NIE to 
know what is needed, to be aware of the legal/political constraints, and to 
find ways to mediate the tension between needs and constraints — i.e., meet- 
ing the needs without violating the constraints. Perhaps one mode of such 
mediation would be for NIE, acting in consort with other agencies facing 
the same tension, to apprise the Congress and other relevant groups of the 
long-term requirements for system building as contrasted to procurement 
approaches to funding, given the current system state and needs. 

In the above discussion, we have touched upon two other issues relevant 
to NIE which are dirrcussed in our analysis: inter-agency coordination and 
internal NIE staffing (especially the question of an internal NIE Research 
capability ) . 
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Although a detailed analysis and discussion would be beyond our 
current seope of effort, our analysis indicates that inter-agency 
coordination and orchestratioii is a key NIE role — both because NIE is 
the lead agency in educational R&D and becaus^^more funding of educa- 
tional R&JD is _ provided put_sJLde^ As we__noted 
earlier^ this may at t me s mean- that .a significant portion of NIE 
efforts may be applied toward a specific critical area even though 
relatively little of NIE'fr budget is applied to this same area. Stated 

"another way concern should be with needs, not only with 
the implications of its budget per se. 

Our analysis also suggests that the way NIE provides for its own 
internal staffing will have a critical effect on the direction and 

.effectiveness of the Institute. For example, the NIE role of .orches- 
tration requires personnel who have skills in orchestration and in 
facilitating collaboration between people and/or between institu- 
tions and agencies • Additionally, NIE will need sone personnel who 
have "political sawy". For another example, our analysis will 
suggest a need to build the Research function and will further sug- 
gest that only by having an internal Research capability will NIE be 
aole to orchestrate the building of the Research functions. 

j^^ might also comment briefly on the issue of NIE "rules of 
thumb" (such as: "only offer a grants competition when a total of 
one million dollars can be provided and when you can fund 25% of the 
proposals submitted") • We would not be concerned with the amount 
and the percentage figures per se. Rather we would note that there is 
an inherent system logic in such a rule of thumb — i.e., it is correct 
that the expectations of the field should not be raised beyond 
reasonable levels of potential for fulfillment. We would further 
note, however, that "rules r^'f thwb" tend, too often to fall into the 
trap of ignoring critical system or function dynamics, conditions, 
needs and requirements. In the above case the danger would be that 
such a grants competition would be used in an area so lacking in 
Excellence that the funding of 25% of proposals at the one million 
dollar total level would tend to trap the Agency into indeed provid- 
ing the funds to low quality, low-success -probability projects. 
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One final comment. Our analysis may at first glance appear overly- 
extended for such a "simple" problem as the Field-Initiated vs. Agency- 
Directed Issue. Our point is precisely that this is not a simple question — 
it is embedded within other more fundamental system issues, and NIE's re- 
sponse will -have system-wide impact • Thus, the only appropriate auialys is is 
a system analysis. Additionally, this analysis has shed li^^^ 
- - issues confronting NIE — a good illustration of the "multi-purpose effects" 
concept we-wlllintroduceiin this report. 

III. REPORT ORGANIZATION AND PREPARATION 

"In. the pages that follow:^ our report will; 

1. describe our method of analysis; 

2. analyze the NIE/field relationship issue from both the.gener ic 
and educational-context perspectives in four major functional 

" areas of educational R/D&I— Research, Development, Dissemina- 

tion, and Evaluation Research; 

3. outline potential implications of the analysis for NIE pol- 
icies and strategies in each of the four functional areas; 

4. provide hypothetical scenarios to illustrate the implications 
of potential policy and strategy decisions of NIE. 

The policy analysis is framed within the overall contextual issue 
of the relationship between a mission-oriented federal agency (NIE) 
and the operative R/D&I community in a specific sectoral context (edu- 
cation in the United States) . The specific issue of Field-Initiated 
vs. Agency-Directed Procurement is treated as a sub-issue of the larger 
Agency/Field relationship issue. 

Reference 

Seiber, Sam D., "The Requirements of a National Educational R&D System" 
in Educational Researcher, December 1975 (p 3-6). 
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I. ISSUE DEFINITION 

1, Initial NIE Concerns 

As presented to us by NIE, the concerns of the Agency focused on 
two questions of procurement policy. Foremost was the question of 
determining the appropriate balance between "Field-Initiated" versus 
»'NIE-Directed" R5D. The Agency had been receiving considerable 
criticism from the field, reflecting the view that too limited a role 
was permitted the field in the design of either broad programs or 
specific procurements. Many of those unhappy with the present charac- 
ter of procurements were suggesting that the Institute commit itself 
to a predetermined funding percentage for field-initiated R5D. The 
expectation was tViat such a predetermined set-aside formula would 
create a larger flow of funds into field-^ initiated work than had recently 
been the case. The National Council on ' ^ : :ational Research, in its 
desire to be responsive to the field, requested a review of NIE procure- 
ment policies and some careful consideration of the question of how 
Agency policy might be developed on the matter of the appropriate 
balance between Field-Initiated and NIE-Directed R5D. 

A second, related question involved determining the appropriate 
mechanisms for procuring Field-Initiated and NIE-Directed work. Of 
particular concern were questions about the nature and extent of 
direction that an agency could justify building into procurement 
mechanisms, specifically with the requirements and language of RFPs 
(as these might constrain project conception, design, execution, etc.) 
and most particularly whether the RFP should be used at all in the 
R5D funding context. 

2. Our Reformulation of the Issue 

In thinking about these questions, we were struck by the rather 
fundamental and broad-ranging implications of the questions raised, 
especially when viewed from the framework of our understanding of R5D 
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systems and processes. In order to deal with these far-reaching issues 
in a meaningful way, it seemed to us that there were some critical 
prior questions to be addressed and clarified. Therefore, we proceeded 
to reformulate the questions of concern to the Institute in a manner 
that we felt would best permit us to shed some light and suggest some 
directions for policy development. 

Our consideration began with recognition of two defining features 
of the Agency's character: first, that NIE is a mission-oriented R§D 
agency; and second, that NIE is the lead agency for federal activity 
with respect to Research and tevelopment in education. Given that 
role, its funding policies would have to be understood in terms of its 
purposes as these impact on the total education sector's Research, 
Development and Innovation (R/D^I) system. What the balance of differ- 
ent types of funding should be and how an agency should relate to the 
field with which it worked would depend on the purposes the agency was 
trying to achieve across all aspects of the R/D5I system. The nature 
of this behavior would need to be fitted to whatever it was necessary 
for NIE to do if it were to achieve its mission in all its R/D5I system 
aspects. Percentage of field-initiated programs and type of procure- 
ment mechanism used could be viewed in this light as indicators or as 
symptoms of Agency/Field behavior rather than as direct policy lever- 
age points. Therefore, it seemed to us, our analysis could be focused 
most fruitfully on more fundamental questions concerning NIE's mission 
and purposes in relation to the field's needs and conditions. Answers 
to these prior questions, if seen as determinants of necessary Agency 
behavior, would suggest what the appropriate Field-Initiated/Agency- 
Directed balances should be at any particular time, and would suggest, 
too, the most appropriate procurement mechanisms for each individual 
case. With this set of assumptions as our starting point, we then 
began to formulate our analytical strategy. 

3. The Relationship of this Analysis to Current NIE Analyses of this 
Issue 

In carrying out our analysis the results of recent in-house NIE 



EKLC 



17 



efforts (specifically the Duffy et al. 1976 memos) were made available 
to us. We deem it appropriate to point out the areas of similarity 
and points of departure between our analysis and the NT:, in-house 
report, 

A. Review and Interpretation of the Duffy Report 

From our pei-spective, the NIE study was a rich, on-target 
discussion but one that was acutely limited by its narrowness of focus. 
We make this criticism on two grounds. 

a) The report lacks an overall R/D5I systems perspective, 
casting its arguments in terms most relevant to the Research function, 
but far less appropriate to the NIE purposes with respect to, and the 
generic issues inherent in, such other functions as Development, 
Dissemination, etc. 

b) The discussion was, in our view, more than necessarily 
couched in terms of a Field/Agency dichotomy (an "us" versus "them" 
perspective). Rather, it will, as we will show, be vital to recognize 
NIE's integral place as a part of the educational R/D5I systenu Despite 
these criticisms we would be remiss if we failed to comment on the 
quality of the analysis within the above stated constraints. 

B. Our Approach 

By contrast, our approach grows out of and builds on a systems 
perspective, with NIE's mission being viewed in terms of its impact, 
as an irte^ral part of the system, on the educational R/D5I system's 
health, functioning and outputs. Further, grov/ing out of this systems 
perspective, and as is inherent in our general analytical method, we 
engage in a broader, more systematic analysis of R/D5I functions auri 
the range of conditions affecting the system. Finally, we note th:it 
procurement is but one of the range of behaviors available to NIE by 
which it can influence the system and that behavior must be evaluated 

in its totality, 

Tnus, the essence of the issue as we see it is: how does NIE 
achieve its purposes through procurements and other Agency actions. 
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taken in consort with and as part of the field? 

4. NIE Purposes 

A. The Multiplicity of Purposes 

Central in determing NIE's proper modes of behavior must be its 
mission in relation to the educational R/D5I system. While NIE can be 
conceived as seeking many individual goals these can be usefully grouped 
under the following general systems dimensions: 

1. Substantive outputs of the R/DSI system (knowledge, products, 
services, etc.) - The systems throughput dimension. 

2. System capacity building (institutions, linkages, personnel, 
etc.) - The level of maturity and capability of the system 
itself. 

3. Affecting the system environment (support, prestige, legiti- 
macy, etc.)- The system environment. 

Procurements tend to be thought of primarily in terms of the 
first of these categories, the direct purchase of R/D5I activities to 
generate knowledge, produce programs, products, etc., or to provide 
services. Occasionally, agencies procure capability-building activi- 
ties directly, as in the provision of institutional support, or the 
funding of training programs or graduate or post-doctoral fellowships. 
But for the most part, procurements are desigajed and managed by 
agency personnel as individual projects or pragrams designed to pro- 
duce specific outputs for the use of the operational system or the 
R/D5I system itself. 

What tends to be overlooked is the exteiit to which these mani- 
festly single-purpose procurements tend to have multi-purpose impli- 
cations: in almost every procurement (or other Agency behavior), more 
than one of these purposes will be involved,Cwhether implicitly or 
explicitly. Thus, the award of a grant to an R§D institution to 
support a specific: project may also have an impact on that institution's 
capacity to perform in the future (e.g. by permitting it to hire 
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additional personnel, by the added experience that may result). 
Similarly, the provision of an institutional support grant may result 
in the conduct of R§D programs whose outputs may not have been speci- 
fically sought but which are of considerable value, and at the same 
time act so as to increase that institution's legitimacy vis a vis 
various of its publics. 

Consequently, it becomes essential for an agency to be very clear 
about its purposes, those entailing system building and affecting the 
system environment as well as the use of system capacity to produce 
substantive outputs. And too, it seems important to develop ^some 
recognition of the legitimacy of latent as well as manifest purposes 
for procurements as well as other Agency actions. 

B. Manifest ap 4 Latent Purpose s 

The legitimacy of latent as well as manifest purposes of Agency 
actions is a point that merits some elaboration. The manifest reason 
for supporting a particular project may have little relevance to the 
real reason, which is latent, implicit, and infrequently made clear 
to members of the R5D community and/or relevant publics. A particular 
project may receive funding not so much because of the immediate 
payoff e.\pected from the project itself but rather because of the support 
it is providing for a certain type or group of graduate students, or 
because it is expected that if a certain Researcher is supported long 
enouc^h he is bound to make very substantial contributions, to the field. 
In such cases, defending a project in terms of its manifest purpose 
may be difficult, but justifying it in terms of long-term capability- 
building needs may be much less of a problem. Or to consider a some- 
what different example, an agency may be subjected to considerable 
pressure to support a particular kind of program, and the pressure 
may be substantial enough to have serious enough ramifications to 
jeopardize achievement of important objectives. In such a case, an 
agency may have little interest in the manifest purpose of a project, 
but may support it for the latent purpose of relieving undue stress 
on the system. 
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The essential point here is that procurements may provide the 
greatest long-range payoff if they are designed with multi-purposes in 
mind, and if Agency 'personnel can design them creatively to serve 
latent as well as manifest purposes. What would seem to be needed, then, 
are deliberate Agency strategies to capitalize on the multiplicity of 
consequences from specific Agency actions, to maximize possible gains 
and minimize possible costs from potential multiple and interaction 
effects across the latent and manifest purposes of given procurements. 

C. Interaction Effects 

This issue of interaction effects is one of the most critical 
points that seems to be overlooked in the development of Agency policies. 
Once an Agency comes to view its behavior in terms of interactions 
among seemingly discrete actions, an entirely different kind of 
understanding emerges of the potentially far-reaching systemic impli- 
cations of individual decisions and policies. Different purposes can 
interact with one another (a point we shall return to shortly) . Pur- 
poses can interact with procurement mechanisms - e.g., a mechanism 
used to procure Basic Research outputs can have major implications 
for long-term capacity building. Purposes and mechanisms can interact 
with contextual conditions, e.g., the state of development of the 
system; a strategy that may have been ineffective a decade ago may be 
highly successful in achieving certain purposes now or ten years from 



now. 



The point is perhaps made mor;t clearly by examining potential 
interaction effects among purposes, both within a single procurement 
and across the totality of procurements made by an agency. A procure- 
ment can lead to the creation of outputs and lead to an improvement 
in the system environment. Or, it can lead to a deterioration in the 
environment if, for example, that particular output is seen as 
offensive to certain key elements. It could also lead to a destruction 
of R/D&I system capacity by, for example, moving critical resources 
away from their most productive areas of application. 

When one examines patterns of Agency actions across procurements 
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i.e., when one considers potential interactions among the discrete 
procurements that make up an agency's ^'portfolio*' interactions of 
an even less obvious nature become apparent. Across programs, the 
outputs may reinforce each other (synergistic effects). Or they may 
counteract each other in the manner of what might be called '*anti- 
purposes" — i.e., taking a specific action in pursuit of one purpose 
may make more difficult the achievement of another purpose. The use 
of RFPs to procure certain kinds of Research, for instance, might 
well have anti-purpose effects if a by-product is turning off the 
best Research talents, suggesting to them that Research funding in 
the field of education is unlikely to be forthcoming without untenable 
constraints. Such effects may be immediate in their interaction or 
observable only in lagged and in second- and third-order manifestations. 
If an agency decides to design procurements that are deliberately 
multi-purpose in nature, it becomes essential for agency personnel to 
have a clear understanding of the kinds of procurement "add-ons" that 
tend to be congruent vs. incongruent with each other, functional vs. 
dysfunctional . 

Portfolio effects may be discernible within institutions as well 
as across institutions. It is common to observe how RSD institutions 
become shaped by the patterns of funding that become available to them. 
If a single agency provides a particularly large share of an organiza- 
tion's total funding, agency actions can have the effect of molding 
or changing the very character of such organizations. 

In summary, then, interaction effects will need to be considered 

in terms of their: 

synergistic effects 

congruency/incongruency with each other 

- lagged (and indirect/second and third-order) effects 

- cumulative effects within and on institutions and personnel 

The essential point is that multi-purpose effects are inevitable. 
The issue is not whether there should be multi-purposes but rather 
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whether they are to be recognized or ignored, and if recognized to be- 
dealt with and capitalized upon or anti-purposes minimized. 

5. The Functional Context of NIE Purposes 

Up to this point, we have considered three sets of purposes that 
can be affected by procurements and other Agency behaviors. We turn 
now to consideration of the second major building block of our analyti- 
cal approach. 

NIE's purposes are achieved through the carrying out of various 
activities that can be categorized by R/D&I functions: Research. 
Development, Dissemination, etc. Achieving the same purposes (e.g.. 
institution building or affecting the system environment) may call for 
different Agency behaviors in relation to these different functions. 
Building Besearch institutions may demand strategies very different 
from those required to build Development organizations. The mechanisms 
that are appropriate for procuring Development products may be quite 
inappropriate for procuring Basic Research studies, and so forth. 
Similarly, the types of skills and experience required within NIE 
to work with personnel and institutions involved in Basic Research are 
likely to be rather different from those required when working with 
those involved in the Dissemination function, n^erefore. it follows 
that the determination of Agency behaviors (of all kinds) are likely 
to be highly dependent on the R/D§I functions with which they are 
involved, and this consideration must be reflected in our policy 
analysis. 

To make such analysis possible, it was necessary for us to select 
a set of R/D5I functions that seemed to be reflective of the bulk of 
the activities that go on within educational R/D§I and that become the 
object of NIE procurement and other Agency activities. With this m 
mind, we selected the following R/D§I functions (or groupings of 
functions)to become the focus of our analysis.* 

* Tn thP R/Dfl systems analysis scheme we use generally, we treat Need 
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A. Research 

Research can vary along a series of multi-dimensional continua, 
generally categorized as going from Basic to Applied. We fully 
recogrize the debate involved in such categorization and the diffi- 
culties involved in the usual over-simplification so implied. Never- 
theless there are, for our purposes in relation to discussing appropriate 
Agency behavior, important potential differences between what is 
required to deal with Basic Research, which is largely involved in the 
search for knowledge for its own sake, and what is required for applied 
work that goes on in relation to well-defined problem areas. We will 
therefore examine these two types of Research as being representative 
of the range of activities with which NIE may become involved. In 
thinking through the implications of our analysis for the particular 
kinds of Research NIE procures. Agency personnel can make the necessary 
accommodations to variations encountered between these two extemes. 
The two sub-functions therefore will be: 

1. Basic Research (the seeking of knowledge for its own sake) 

2. P roblem Focused (or Applied) Research . As used here and 
later in this analysis "problem" refers to a social or 
practice-centered problem rather than to the kind of 
intellectual or discipline-based problem that is central 
to Basic Research. 

B. Development 

Whereas what we are calling Problem-Focused Research is oriented 
toward problem areas within education. Development work tends to be 
focused on the design and elaboration of products, processes, programs, 
procedures, practices, etc. that attempt to deal with identified 
problems or needs. For simplicity of usage, we will generally use the 
term "products" to describe the outputs of the Development process. 
However, it should be understood clearly that we have the full array 
of Development outputs in mind — programs, procedures, strategies, 
practices, etc. as well as the narrower category of outputs generally 
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thought of as "products." 

r n-i .;.;emination/Iinp lemen tation/Utilization 

The functions within this cluster are typically treated separately 
in R/D5I systems analyses (at least as between Dissemination and the 
other two) . We will link them in this analysis since this is a 
characteristic of the educational R/D§I system: its relatively low 
level of maturity and the general absence of institutionalized integral 
User change agent functions make this a necessity. The focus of our 
analysis will be placed on the Dissemination function, since it is here, 
in system-level linkages (rather than within the User setting that 
shapes Implementation/Utilization) . that NIE efforts can have more 
substantial impact. However, it should be noted that any future ex- 
pansions of our analysis should involve specific detailed analysis of 
the Implementation/Utilization functions. 
D. Evaluation/Policy Research 

Evaluation and Policy Research are often grouped together. In 
education this seems particularly fitting since Policy Research in 
education so often involves one or another form of Evaluation Research. 
However, there are important differences between Evaluation Research 
and Policy Research, and they may require some extended separate 
treatment. Since it appears that the bulk of NIE's efforts in these 
areas are more directly involved with Evaluation Research, we will 
focus on this function in our analysis. Again, further work could 
expand on the specific issues related to Policy Research. 

6. R/DSI System Context 

We have seen up to this point in the analysis that achieving the 
same Agency purposes may require somewhat different Agency behaviors 
in relation to different R/D§I functions. But beyond this, for R/D§I 
functions and their generic requirements to be understood in terms 
that seem congruent with concrete empirical reality, it becomes im- 
portant to see each function within^a total R/D5I system context. 



We define an R/DSI system context as the joirt interaction of 
three elements: 

1. the R/DSI functions , as described above • 

2. operating conditions within a particular R/D5I system 
(e.g., the maturity of the system; the types and quality 
of personnel available; the types and quality of the insti- 
tutions available; the state of development of the know- 
ledge/technology base; the nature of the information 
systems and flows among key elements in the R/D5I system 
journals, invisible colleges, conferences, etc. etc.; the 
very nature of the innovations involved). 

3. the system environment (e.g., the political/social environ- 
ment, especially its supportiveness or lack of support for 
the system; the economic environment; the nature of the 
knowledge base of the field -- whether it is a natural- or 
social-science base; etc.). 

Therefore, in determining appropriate Agency behavior for achiev 
ing a certain purpose it becomes vital to be cognizant of both the 
system function to which it applies and the manner in which the generi 
characteristics of a given function are mediated by systemic and 
environmental conditions. It is this joint effect that we term the 
R/D&I system context. 

7. The Spectrum of Agency Behavior 
A. Types of Behavior 

While an agency such as NIE can engage in a wide variety of be- 
havior in relation to its purposes, these can be usefully grouped for 
analysis under three headings: , ^ 

1 . Procurements 

An agency's use of the funds available to it to procure specific 
outputs, institution building, etc. is generally seen as the prime 
forum for Agency action. 
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2. Non« Procurement System Behavior 

The potential range of Agency behaviors is -'not necessarily limited 
to procurements. It may also play a potentially important role in the 
system through a variety o£ other kinds of actions. These may include- 
working with other agencies so as to achieve synergistic and multiplier 
effects from joint (additive and/or supplementary) activities; having 
its personnel play an active role as members of the R/D§I community — 
as Researchers, as participants in conferences, as influences on 
thinking in various informal interactive modes, etc.; by making in- 
formation available; etc. 

5. NIE Internal Actions 

How an agency relates to a field will be inportantly determined 
by what goes on within the agency. Specifically, the strategies and 
modes of behavior required will be constrained by the extent to which 
NIE has the number and type (skills, experience, stature) of personnel 
needed and the organizational and budgetary structures that pennit 
appropriate behaviors . 

B. Types of Strategies 

Within the above three areas for NIE behavior, there remains a 
wide and multi-dimensional variety of behaviors in which an agency 
can engage. These can be condensed along three dimensions: 

1. Degree of Agency Control 

NIE might see the need to maintain a greater or lesser degree 
of control over what goes on in the field, in terms both of extent 
(level) and of domain (program selection, methods used, personnel 
involved, nature of relationships, forms of reporting,etc.) . 

2. Degree of Agency Involvement 

NIE might choose to be more or less directly involved in what 
wa§ occurring in the field— for example: NIE participation in the 
variouts functions (e.g., undertaking various types of Research, 
Dissemination, Evaluations, Policy and R§D system studies); planning 
for and monitoring of the R/D§I system; determining what work to carry 
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out internally and what activities to procure through external con- 
tracts or grants. 

5. A Strategy Continuum 

Seen the above ways, it is possible to array very roughly the 
types of strategies by which an agency can relate to various aspects 
of a field. Such an array could be the following: 

- Initiate activity for a field 

- Supplement whac is already going on 

- Modify what is already going on 

- Select from among what is already in a field 

- Educate the field to operate differently 

- Mediate external pressures on specific field elements or 
programs ' 

- Integrate programs, institutions and systems to be found 
in the field 

^ Cooperate with other programs (e.g. in other agencies) or 
with field programs 

- Facilitate activity already going on 

- Execute activities initiated by the field 

- Evaluate activities that have occurred in the field 

- Monitor what is going on. 

This array, while admittedly rough, represents, jointly, a 
diminution of Agency control and involvement in what is going on 
in the field, in both pragmatic and systemic terms. The order is 
not, however, especially important. To the extent that it does 
array a variety of possible strategies, it demonstrates a richness 
that goes well beyond the simple Field-Initiated vs. Agency-Directed 
continuum. From our perspective, the Field-Initiated vs. 
Agency-Directed continuum is likely to be most meaningful in 
relation to individual projects and programs rather than system- 
based purposes, and in relation to those functions in which programmatic 
activities can be separated easily from system activities, at least 
in the short run, as in the case of Research, especially Basic Research. 
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(And even then the extent to which the FIS vs NIE continuum applies 
seems related to the extent to which Agency personnel focus on their 
own individual projects rather than the system as a whole.) The Field- 
Initiated vs. NIE-Directed continuum is likely to be far less meaning- 
ful for such inherently systemic functions as Dissemination, or for 
system building purposes. Seen in this way a determination of the 
proper proportion of NIE's budget that should go to Field-Initiated 
activity could only be made in relation to NIE's agenda, as it played 
itself out with respect.to. the Agency's purposes, as manifested in 
the various R/D5I functions (Research, Development, Dissemination, etc.), 
under the prevailing contextual conditions (personnel, funding, 
maturity of the system etc.). As an overall index it would therefore 
not seem to provide much meaning, and hence not represent a proper 
actionable policy criterion. 

8. A General Analytical Model 

As implied above, our analysis will involve the specification of 
the behavior appropriate to the achievement of Agency purposes in the 
context of the functional, systemic and other environmental conditions 
that prevail. Diagrammatical ly, this can be illustrated as in Figure 
1. This model also indicates that NIE's piirposes will themselves be 
influenced by what is going on in the educational R/D§I context and 
that in turn this context will be importantly influenced by how NIE 
does actually behave, as an integral part of the educational R/D§I 
system. 

II. METHOD OF ANALYSIS 

1. General Methodology 

A complete analysis would require examination of how the inter- 
action of NIE's purposes, as manifested in the R/D§I functions , and 
mediated by the syst emic and environmental conditions, determine 
appropriate Agency behaviors and consequently strategies in relation 
to the field. The above statement would imply at least a four 
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and the R/D&I System Context 



H 



X/ 



dimensional analysis. As a simplifying step, we have elected to 
consider each of the R/D§I functions separately in relation to the 
combination of R/D^I systemic and environmental conditions, as shown 
in Figure 2, 



R/DSI Environmental and Systemic Conditions 

R/DSI Function (Appropriate NIE Behaviors/ 

NIE Strategies) 



Figure 2. Simplified Analysis Matrix 

The cells of this matrix are the appropriate NIE behaviors which can be 
built into strategies that the Agency could pursue. 

Having conducted the individual analyses for each R/DSI function, 
it becomes possible to consider the implications for NIE strategy 
across the functions. Finally, the strategies can be converted into 
scenarios in which patterns of hypothetical, or actual past or con- 
templated NIE behavior, are analyzed to suggest likely impacts if 
implemented as originally formulated, or as reformulated in alternative 
ways that take into account some of the points we have tried to under- 
score in our analysis. In the final analysis, one could conceive of 
converting NIE's total programmatic agenda into an integrated set of 
scenarios, determining the various patterns of appropriate Agency 
behaviors (a fallout of which would be an estimate of the percentage 
of all procurements that would be Field-Initiated—though as we have 
stated this would not be an index of great significance in an of itself), 
and suggesting likely impacts. 

2. Specific Analysis Method 

Within the framework of the above general approach, the procedures 
to be followed will be as follows: 
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A. Functional Analyses 

Each selected R/D§I function (Research--Basic and Problem-Focused-- 
Development, Dissemination, Evaluation) will be analyzed in the follow- 
ing manner: 

1. Generic Characteristics of the Function 

We will begin each functional analysis with a review of those 
generic issues inherent in the function which are likely to be of 
relevance to educational R/D5I and have implications for NIE behavior. 
A complete generic review of each function would inevitably deal with 
many issues that are of relatively lesser concern in education at this 
time. Given the constraints of time, volume and salience implicit in 
a policy analysis, we have attempted to be judicious in our selection 
of issues to be discussed. 

2. The Educational Context of the Function 

We then consider the current state of affairs in the educational 
R/D5I system as it relates to carrying out this particular function. 
We examine contextual issues pertaining to the state of development of 
the relevant knowledge base, the institutional base, the personnel 
base, the climate of support for funding its activities, etc. As before, 
the implications for NIE behavior are drawn. Where feasible, this 
section concludes with some general guidelines for NIE's operating modes 
and strategies . 

5. Implications for Agency Behavior 

In this final section of each functional analysis we attempt to 
summarize the requirements for NIE behavior in relation to the parti- 
cular function in its present context, building up some recommendations 
for Agency strategy. 

B, Cross-Functional Analysis 

In this critical chapter, we take the analysis and strategy- 
building a step higher by attempting to draw the cross-functional 
implications for NIE at a total Agency level. 
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C, Scenario Analyses 

Two illustrative scenarios are then analyzed. Each scenario 
consists of a description of a hypothetical NIE behavior 
(e.g., a particular procurement program, its objectives, the manner 
of its implementation), an analysis of its wide implications (if any) 
and likely impact, our recommendations as to what changes might 
(or should) have been made, and the likely consequence of these 
recommendations. The thinking behind this process is depicted 
graphically in Figure 3. 

A specific NIE action (usually a procurement), with its intended 
purpose, would result in some consequences. The effects or impacts 
would be a consequence of the NIE action interacting with contextual 
conditions, and would have to be understood in terms not only of the 
intended and manifest purposes of an action, but also any other 
(possibly latent) purposes. These impacts would be evaluated and 
appropriate strategy alternatives recommended. Such recommendations 
would lead to NIE actions involving procurement, non-procurement and 
internal NIE behaviors, in relation to the whole range of possible 
purposes. Then, in turn, the effects of these behaviors would be 
analysed. 

The scenarios are intended to suggest an analytical approach we 
view as appropriate for internal Agency use in designing procurements 
and relating procurement strategies to other, possible non-procurement 
courses of action. The strength of this approach, we would argue, is 
the manner in which it orients Agency personnel toward system-level 
thinking: 

1. It requires the analyst to think in terms of the 
multiplicity of purposes implicit in procurements and 
other Agency behaviors, and suggests the legitimacy of 
designing courses of action in terms of latent as well 
as manifest purposes. 

2. It requires consideration of interaction effects among 
purposes, between purposes and mechanisms, and between 
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purposes and generic characteristics of R/D5I functions 
as these are mediated by systemic and environmental 
conditions , 

3. It calls for Agency personnel to estimate potential 
impacts of contemplated courses of action on key »dimens ions 
of the system, 

4. It suggests a series of questions that enable the analyst 
to generate alternative courses of action with potentially 
different impacts, and to use these alternatives, along 
with their own estimates of potential impacts, to reformu- 
late and refine contemplated strategies • 

We offer the analysis which follows as a first-cut at what we 
believe can evolve over time into a highly useful approach • 

REFERENCES 

Duffy et al ., 1976. 

We had access to several internal memos, in variovis states of their 
development, in August and September 1976, including: a) an August 2, 
1976 memo written by Susan Duffy and Noel Brennan entitled "Status 
Report on FIS"; and b) a somewhat later set of materials providing 
working definitions of Narrow and Broad FIS, elaborating the relevant 
dimensions of each, and considering some of the implications for NIE. 
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Research can be viewed as being essentially of two types: 

1 . Basic Research: Research to produce knowledge for its 
own sake. 

2. Problem-Focused Research: Research seekin;^ to produce 
knowledge applicable to the solution of a specified 
problem (commonly known as Applied Research). 

In this section, we will examine Basic Research. In the next section, 
we will examine Problem-Focused Research. 

BASIC RESEARCH: KNOWLEDGE FOR ITS OV/N SAKE 

I. GENERIC CHARACTERISTICS OF BASIC RESEARCH 

The characteristics of Basic Research have been well and frequcnti/ 
discussed in the literature. I'/e will here attempt only to highlight 
briefly and call attention to some of the particular characteristics 
that are of some importance to this analysis and which bear repeating 
by way of introduction. 

1. Uncertainty and Unpredictability 

The primary characteristic of Basic Research is its uncertainty 
and unpredictability. Foi" example: 

1. It is very difficult to predict the form, the type, the 
timing of outputs--or even of the inputs that Will be 
required. 

2. In the long run, what are later seen as the ''most important" 
benefits of Basic Research were not even initially visualized, 
but rather resulted from spin-offs of the Basic Research or 
from findings that arose in unexpected and unplanned-for 
turns and byways in the Research process. 

3. Indeed, the very definition or specification of what constitutes 
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a "Field of Research" tends to change over time. 
A discussion of the uncertainty and unpredictability of Basic 
Research could be continued for some length. What is important here 
is to recognize some potential implications for NIE. 

1. The Basic Research process calls for a high degree of 
creativity from Research personnel. The specific 
implications of this fact will become more obvious later 
in our discussions where the personnel involved are a 
significant consideration (e.g., in detenaining a 
"minimiim critical mass" of talent) . 

2. Long-range program planning is essentially meaningless 
on a project-by-project basis. Most particularly, it 
seems difficult if not impossible to program the kinds 
of sudden, major "breakthroughs" that are sometimes 
sought. Indeed, in some fields or in some projects, 
such "breakthroughs" may never occur. Rather, progress 
may be incremental, developing through an accumulation 
and extension of knowledge over extended periods of 
time. 

3. Since it is relatively difficult to know where and when 
useful outputs will occur, the primary need in Basic 
Research is for as much high quality activity as possible. 

4. It is important to be sensitive to what is happening in 
a field and how a field is changing, as contrasted to 
thinking in terms of programming the changes in a field. 

2. Time Frame for Outputs 

Basic Research has both a short-term and a long-term frame. In 
the short term, the result of Basic Research is essentially to in- 
crease the total base of knowledge , i.e., to produce knowledge that 
is in turn used and built upon by other Basic Researchers. It is 
generally only over very long periods of time (e.g., 50 years) that 
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the Basic Research results in practical application . Thus, what is 
done in the present, is done for the future. What is lacking in the 
present is a result of omissions of the past* 

5, The Rate at Which a Basic Research Area can be Built Up 

Two points must be made about the process of "building up" a 
high quality Basic Research area. First, a high quality Basic 
Research area cannot be built up very quickly. The cumulative 
development of a total base of knowledge is generally a long-term 
process (as we have just noted). And it takes a considerable amount 
of time to both train competent Researchers and develop a community 
of Researchers who are committed to a particular field. 

Secondly-, the rate at which a Basic Research area can be 
developed is dependent upon the number of competent Researchers and 
centers of excellence already existing in the field. If there are 
only a few competent Researchers and centers of excellence, the 
rate at which the total base of knowledge can be exi^anded is rather 
significantly limited. The training of new personnel who are 
committed to the field must be done by the currently available leading 
Researchers within the context of existing centers of excellence. Thus, 
the fewer such centers of excellence, the slower will be the rate of 
increase in the total number of competent, committed Researchers. 

The above considerations have some significant implications for 
the funding in a Basic Research area. I'/hen the existing quality 
Research base is small, pumping large amounts of money into Basic 
Research would tend to have very little constructive or meaningful 
impact in either the short terra or the long term. Instead, 
the impact would tend to be the generation of a large amount of 
lesser quality activity, much of which would simply be unproductive 
and would disappear (for lack of commitment) when the funding was 
withdrawn. 

Beyond the setting of some upper and lower boundaries, there 
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are virtually no useful guidelines for determining the "optimal" level 
of funding for Basic Research. Rather, funding becomes essentially 
a matter of faith related to the availability of funding. It is not 
even very helpful to make analogies to other fields because: 

1. These other examples were themselveb based on this kind 
of faith and/or his'torical extension. 

2. The existing situation and opportunities can vary so 
dramatically as to make projections very difficult. 

An upper boundary for funding would be determined by the rate 
at which funds can be usefully absorbed by the existing centers of 
excellence- -i.e., the rate at which they could increase the number 
of graduate students and/or take on additional Basic Research programs. 
The lower boundary for funding might be determined by the minimal 
funding necessary to protect the existing centers of excellence (or, 
if necessary, to create such centers) . In setting both the upper 
and lower boundaries, consideration should also be given to funding 
being provided by other funding sources (e.g., other federal agencies, 
foundations) . 

Between the upper and lower boundaries, there would tend to be 
a rather wide range of levels at which funding could potentially have 
significant impact. It should also be noted, of course, that these 
upper and lower boundaries might (for practical purposes) be further 
constrained by economic, political or other non-Research considerations. 

4. Some Other Fundamental Issues 

A. The Criterion of Excellence 

The criterion of Excellence is central and predominant as a basis 
for decision making. Ideally, at least. Excellence is the key selection 
criterion for programs and projects to be supported, funded, or even 
permitted to take place (e.g., by allowing access to scarce facilities, 
etc.), and for the support of institutions and personnel. 
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B, Competing Centers of Excellence 



However, where a Basic-Research field is relatively immature-- 
where it lacks cohesion and consensus on the definition of Research 
areas, clarity on the standards that define Excellence, a well 
developed social structure that establishes a picture of who the 
field regards as authoritative, etc. --the choice among tlibse centers 
of excellence that do exist becomes that much more difficult. Under 
these circumstances, funding agencies are faced with a choice of: 

a. spreading resources around to many centers of excllence; or 

b. "'placing their bets" on a limited number of such centers. 

C. ''Minimum Critical Mass" 

The resolution of choice among competing centers of excellence 
requires (at least) the introduction of another premise concerning 
Basic Research — namely the need to maintain a '"minimum critical mass'' 
of effort. This premise holds that because of the uncertainty and 
unpredictability in Basic Research, because of the need for a creative 
interplay between a number of persons with different perspectives 
(sometimes disciplines, etc.), and because Basic Research is a 
''building upon,'- generally incremental process, there must often be 
a certain "critical minimum" number of Researchers interacting with 
each other if Basic Research is to be productive within a given time 
frame. This minimum critical mass may vary across fields and is most 
particularly applicable to empirically based work. Observations of 
prior successful patterns can permit some estimations to be made. 

Taking together consideration of limited funding, the existence 
of several competing centers of excellence and some notions of minimum 
critical mass, a funding agency might consider a strategy of "placing 
bets" on a limited ntunber of centers of excellence as being superior 
to spreading its limited funds around to many centers of excellence. 

S. A Possible Set of Selection Criteria 

Summarizing from a number of points in the discussion thus far, 
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we suggest the following as a possible set of criteria for selecting 
Research areas for funding: 

1. Which disciplines (or sub-disciplines) have a potential 
for making a contribution in the long run? 

2. Which areas are already being well -supported by the 
Agency or by other funding sources? 

3. Which areas have the sufficient seeds of excellence to 
build upon? 

After having made a determination of these first three criteria, then 
a fourth criterion may be added: 

4. Which of the areas have the best potential for building 
long term capability for the Basic Research function? 

II. BASIC RESEARCH IN EDUCATION 

1. Weakness of Central Core of Educati onal Researchers 

The central core of Basic Researchers, those committed to and 
devoting the bulk of their careers to educational Research, seems to 
be particularly weak in this field, especially when compared to 
Basic Research in most other fields. In education, this core group 
tends for the most part to be located in schools of education, in 
departments focused on each of a number of derivative disciplines 
(e.g., educational psychology, educational sociology). Consideration 
of these settings as the primary institutional bases for Basic 
Research in education suggests that there are relatively few centers 
of excellence and a great deal of mediocrity. Also, when one makes 
the distinction between numbers staffing these departments and numbers 
carrying out significant amounts of Basic Research, the relatively 
small size of this core, and its scattered condition, become apparent. 

A somewhat different picture emerges when one examines Basic 
Research relevant to education carried out in discipline-based uni- 
versity departments (e.g., departments of Psychology or Sociology). 
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Here, there is considerable Excellence, and these disciplines often 
provide valuable inputs. and contributions to the education knowledge 
and technology base. The problem, however, is that the commitment 
of discipline-based Researchers to the field of education tends to 
be variable and shifting over time. After all, their primary com- 
mitment is to their discipline, and education takes a secondary role. 

Taken together, these two conditions create an educational 
Research community that tends to be unstable and amorphous, thus 
complicating the problem of relationships and of maintaining 
communication flows among the parts of the system. What this implies 
for iNIE is a major system building need in order to (a) create the 
vital, stable core of high quality Researchers and (b) facilitate 
and sustain interdisciplinary communication and collaboration. 

2. General Climate 

The general climate surrounding Basic Research in education has 
not been supportive. Basic Research in the social sciences is not 
held in high repute by the public or its representatives in Congress. 
In education the problem is intensified by the difficulty even 
Researchers have in pointing to more than a handful of significant 
developments that are traceable to Basic Research in the field. The 
consequence has been low prestige--complicating efforts to attract 
strong Research talent to the field--and low political support, 
unreliable funding, and lack of continuity in funding emphases- 
complicating the problem of sustaining the work of those strong 
Researchers who have been attracted to the field. Such conditions, 
then, have tended to limit the quality of relationships between NIE 
and the Researchers, and have acted as a constraint on building the 
needed central core of educational Researchers. 

5. The Interdisciplinarity of Education 

Educational Research tends to require an interdisciplinary attack. 
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yet much of educational Research tends to be discipline-focused. 
This issue adds to the difficulty of developing and finding con- 
sensus and hampers communication between the sub-groups in the field. 
Further, the communication mechanisms in educational Basic Research 
are very diffuse. Lacking for the most part are either the in- 
visible colleges or core journal mechanisms that typically structure 
information flow in other Basic Research field and would simplify 
information searches across discipline lines. The relevant journals, 
both those from the disciplines and those within education, are so 
numerous and so inadequately abstracted that the curaulative develop- 
ment of a relevant interdisciplinary knowledge and technology base 
seems difficult to envision unless a lead agency such as NIE inter- 
venes and takes some facilitative actions. 
4. The Funding Sources for Education Research , 

Given the multi-disciplinary nature of educational Research, 
and the rather substantial number of relevant disciplines, the single 
most prominent policy-relevant feature of the funding of Basic Research 
in education would seem to be the existence and use of multiple funding 
sources. Looked at in total, NIE's potential contribution in dollar 
terms is relatively small. This fact puts a premium on the require- 
ment for NIE to be aware of and stay in close touch with the various 
sources that do provide funding for areas of work relevant to education. 
In this way, NIE can effectively use its resources through strategies 
of gap-filling, attempts to pool and coordinate resources across 
agencies, etc. And given NIE's position as the lead agency for 
educational Research and Development, this leadership/ coordination 
role seems central to its mission. 
IMPLICATIONS FOR THE ROLE OF NIE 



Procurement Behavior 

A. Facilitative Orchestration 

Given the conditions we have been considering, the role of 
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NIE would clearly seem to be to orchestrate the various elements 
in the system. However, in contrast to the kind of orchestration 
we will describe later for the Development function, in Basic 
Research the kind of orchestration that would seem most appro- 
priate would need to take place through a combination of: 

1. facilitating the quality activities that are already 
existing in the field (i.e. seek out and support 
Excellence); 

2. selecting the Research areas with which to work; and 

3. being responsive to the shifts and changes that are 
developing in the field. 

Such a strategy of facilitative orchestration would require 
a close NIE relationship with the field— ongoing, one-to-one 
relationships between Agency staff and Basic Researchers in 
the field. This kind of orchestration cannot really be 
achieved through the use of advisory panels representing the 
diverse perspectives of the Basic Research community, even 
the more creative use of long-term panels for shaping and 
monitoring broad programs. Facilitative orchestration re- 
quires certain kinds of internal staffing in the Agency and 
a collaborative mode of Research management that this kind of 
staffing makes possible. We shall return to these points 
shortly. 

B. System Building 

We have emphasized throughout our analysis the need for NIE 
to play a system-building role in managing its Basic Research 
program. We have suggested at several points that a key criter- 
ion in considering contemplated Agency actions with regard to 
this function should be estimating likely consequences for the 
development of long-term Basic Research capability--a building 
process that requires lengthy time spans and is constrained 
by the scale and state of development of existing centers of 
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excellence. If NIE accepts this role, then in project selection 
this might mean accepting the legitimacy of latent system- 
building purposes in procurement, and recognizing that the sys- 
tem-building consequences of a given project are likely to be 
of greater importance than the project itself or its anticipated 
substantive, output. Once an agency accepts a system-building 
role, and is operating in a function such as Basic Research where 
system-building activities cannot be planned or carried out in- 
dependent of the field and Agency actions can be of only the 
facilitative/collaborative type if they are to be productive, 
then the Field- Initiated vs. NIE-Directed issue loses its meaning 
and forces consideration of a broader and more complex array of 
options. 

C. Implications of Levels of Consensus in a Field 

Different Basic Research areas are likely to vary in levels 
of consensus among Researchers in the field—as to key questions 
in need of answers, adequacy or appropriateness of different 
methodologies, etc. Under conditions of low consensus, the role 
of a funding agency would seem to be to work with the field 
rather than be directed by it— be fairly active in molding and 
selecting from what the field has to offer rather than just re- 
sponding to scattered field-initiated proposals. On the other 
hand, under conditions of high consensus, such activity on the 
part of the Agency would seem tc be less necessary. Thus, the 
Agency could operate in a mode in which one was responsive to 
field initiations. This perspective is somewhat different from 
that presented by the Duffy et al . 1976 memos, it is also worth 
noting in comparison to the Duffy et al_.memos that under condi- 
tions of high consensus there would not seem to be the need for 
RFPs that seemed to be suggested there. 

D. Process Mode of Management 

Up to this point, we have focused our attention on the early 
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stages o£ Agency management of Basic Research, i.e., the selection 
of Research areas, institutions, and projects to fund. We have 
not as yet considered how an agency monitors and manages com- 
ponents of its Basic Research programs after these selection 
decisions have been made. A few comments would seem to be in 
order here. 

Given the unpredictability and uncertainty inherent in the 
Research process, tight monitoring of Basic Research programs in 
accord with predetermined output specifications is clearly in- 
feasible. A more reasonable alternative than tight management 
of the Research product would seem to be monitoring of the 
Research process . However, given the nature of Research and 
especially of Basic Researchers, bureaucratic controls of the 
Research process would hardly seem to be in order. Instead, the 
problem resolves itself if the Agency has an internal Basic 
Research staff that is itself involved in Research and is func- 
tioning as an integral part of the field. Under such circumstances, 
NIE becomes part of the quality control mechanisms operative in 
the field, and may be acting both to insure the needed degree of 
quality control on given pieces of work while also stimulating 
the field's development of self-controlling mechanisms that make 
an Agency role in quality control less and less relevant over time 
as the system matures. 

2. Considerations Internal to NIE and Non-Procurement Activities 
A. NIE Staffing 

The procurement behaviors we have been considering--facili- 
tative orchestration, system building, and process management — 
all require the kind of close NIE relationship with the field 
that suggests a rather particular type of NIE staffing. Speci- 
fically what is implied is staffing an Institute Basic Research 
unit with personnel who possess considerable substantive com- 
petence in the Research areas with which they might be working. 
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It suggests too that these NIE Research personnel might them- 
selves be involved in the Researc. process. This would seem 
to be vital if such personnel were to be sensitive to the shifts- 
and revolutions taking place in the field and to be able to 
discriminate substantive advances and quality as compared to ab- 
berent shifts and poor quality. 

Given those circumstances, it is quite likely that the skills, 
experience and interests of NIE personnel would strongly in- 
fluence NIE decisions about which Research areas NIE would fund 
or support. While this might be of some concern, it need not 
be considered a major problem. Given the large number of poten- 
tially relevant Research areas, and the virtual impossibility 
of determining which of these are most likely to be productive 
for the future, there is no "right" answer to which Basic Research 
areas an agency such as NIE should support. There is no reason 
why an agency should not be opportunistic in selecting areas to 
support--taking advantage of the capabilities of its personnel, 
and permitting them to work with the field in the Research areas: 
Ca) in which they find the most that interests and excites them, 
the most of what they view as productive work going on; and (b) 
those Research areas they are most strongly qualified to work 
with (and in) themselves. 

What we are suggesting here is the need for NIE's Basic 
Research programs to be staffed by personnel who are themselves 
highly competent Researchers in any of a number of Basic Research 
areas of relevance to education. And further, we are proposing 
reviving the idea initially included in NIE's structure and 
operations in the form of its Basic Studies unit— the notion that 
NIE should support an in-house Basic Research capability — not so 
much because of the Research they will carry out per se, but 
because of the manner in which this could permit NIE to have a 
totally different relationship to the field. 

NIE Basic Research personnel, under these conditions, would 
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likely have an intimate knowledge of a number of relevant Research 
areas and the Researchers who carry out the work in those areas— 
who they are, what they are doing, what they have done in the 
past, what approaches they use, vMt r,a.t^cs of the relevant know- 
ledge/technology base of the fielH they draw on. They would 
know how to use the informal communication mechanisms of a Re- 
search area, or where and how NIE should start in any contemplated 
efforts to facilitate the development of invisible college mech- 
anisms or other informal communication channels. And equally 
important, they would have a very clear sense of whatever standards 
exist for judging Excellence, and too, have a sense of the sorts 
of questions to be considered in judging Excellence or in putting 
panels together to guide such judgments. 

Perhaps most critical of all, with such personnel on NIE's 
staff, the NIE vs. Field issue would lose much of its meaning. 
Rather than NIE and the field confronting one another as discrete 
entities with often different interests, the boundary lines be- 
tween the two would be more difficult to define and NIE could 
more adequately function as not simply an agency responsive to 
the field but as Cin fact) an integral part of the field. 

How can an agency such as NIE attract the kind of Basic 
Research talent we have been suggesting is needed? The problem 
is by no means a simple one. On the one hand, the Agency wants 
to attract strong Research talent. But it is unlikely that such 
personrel would be willing to leave Research settings for Agency 
positions for more than short stints (unless, of course, NIE 
could provide an attractive Research environment that could both 
attract and hold such talent--a strong argument in favor of in- 
house Basic Research units). On the other hand, long-term com- 
mitments to the Agency would seem essential if the familiarity 
built up in working with Researchers in a given area is to have 
long-range utility for the Agency. A certain amount of inter- 
institutional mobility may be inevitable, and given that, perhaps 
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could even be capitalized on, e.g., by developing exchange 
programs for two-way flows of personnel between the Agency and 
the field, with exchanges entailing long-term roles with in- 
herent continuity regardless of the institutional affiliation 
of an individual at a particular point in time. Joint appoint- 
ments and collaborative Research are other possibilities. We 
offer these as merely illustrative options. Our basic point is 
that such seemingly internal Agency matters as staffing have 
significant implications across the whole range of Agency actions, 
and may severely constrain Agency options. 

One point of caution should be underscored about the kinds 
of personnel NIE should and should not have carrying out these 
activities. We would caution strongly against the kind of very 
independent "stars" of the field who are likely to distort the 
field to their own image of where the field should be rather than 
facilitating and working with the field. This does not mean that 
there are not some kinds of "stars" who are also good facilitators 
and collaborators. Rather, we are emphasizing the danger of 
selecting the kind of "star" who would tend to pull the field 
only into his or her own image, and would thus tend not to be 
^sensitive to and supportive of other areas of significant activity. 
It is important to recogni::e that there is a considerable differ- 
ence between the role (and hence the kind of personnel) needed in 
a Research institute that is concerned primarily with conducting 
Basic Research and the rather different role needed in a coordina- 
ting funding agency that is promoting Basic Research, as is the 
case for NIE. 

B. Non-Procurement Activities 

If NIE had the kind of staffing we have been considering, the 
' Institute could carry out a number of critical non-procurement 
activities vital to its leadership role in an immature system. 
At the very least we would expect such NIE personnel to 
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carry out such activities as: conducting Research of their own, 
perhaps in collaboration with Researchers in the field; develop- 
ing very clear notions of what quality work is going on, and 
how it could be facilitated; spending a fair amount of time 
travelling to stay in touch with people in the field, and remain- 
ing in touch informally by telephone, etc.; attending or even 
sponsoring conferences and seminars; working with professional 
associations; and perhaps even facilitating the development of in- 
visible colleges and various other coiranunication and quality 
control mechanisms so essential to the cumulative development 
of high quality knowledge/technology bases. 

Beyond this, NIE personnel might well carry out such critical 
tasks as: mapping Research areas; providing critical syntheses of 
the state of the relevant knowledge and technology bases; re- 
viewing what work is currently going on where, what approaches are 
being used, how these relate to state-of-the-art needs and what 
kinds of work still need to be done, what opportunities exist for 
what kinds of talent, etc.; etc. The preparation of such annual 
review documents would necessitate close NIE staff interaction 
and commtinication with the field, and could easily be the wedge 
that would enable th^ field to see NIE as an integral part of 
its functioning and a critical facilitator of its development. 
But only, of course, if these reviews suggested thinking reason- 
ably congruent with the best thinking in the field— again sug- 
gesting the need for close collaboration in the development of 
such reviews. 

Such reviews can serve a number of purposes, not the least 
important of which might be use as a mechanism for attracting 
strong Research talent in certain areas. If such reviews were 
distributed widely, and were sent especially to a carefully 
selected list of capable Researchers whom one might want to attract 
to work in the field of education, the recruitment and capacity- 
building potential of such mechanisms might be considerable. 
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This suggests a non -procurement behavior the Agency might con- 
sider, with considerable long-range implications for procurements 
as well as other aspects of Agency action. This is by no means 
an original idea — NSF has been doing this with some success for 
years . 

Mention of NSF points to one additional kind of non-procure- 
ment activity NIE should be engaged in — i.e., working to bring. T.: 
about some degree of coordination among the various governmental 
and private ie.g., foundations) (Sources of funding for Basic 
Research relevant to education!, ' '^NIE might use its position 
as lead agency for Research and Development in education: a) to 
develop communication channels among funding sources; b) to 
develop analyses of the cross-agency funding patteni C^.g., 
where there are areas of overlap, where there may be gaps); 
and c) to suggest opportunities for pooling of resources where 
the potential for synergy among projects may exist and/or where 
such pooling of resources would seem to increase the likelihood 
of greater overall impact. 

C. Relationship between NIE Staffing, Non-Procurement Activities, 
and a Wider Range of Procurement Options 

To illustrate how the kind of staffing we have been consider- 
ing and the types of non -procurement activities this staffing 
makes possible can impact on Agency procurement options, we 
borrow another example from NSF. If staffed with personnel 
having the proper credentials, we could envision NIE pursuing 
an interactive relationship with the field. An Agency staff 
member might carry out a number of intensive discussions over a 
certain period of time with some of the strong Research talent 
that exists within the field or in other relevant areas. These 
discussions might focus on the kinds of work seen as needed, 
the state of the art for carrying out such work, and what portion 



of all this a particular Researcher would want his own organiza- 
tion to carry out. At some point along the way, some of these 
Researchers might be approached to prepare brief statements-- 
the outlines of thinking oriented in the direction of formal 
proposals. (Or such initial statements might be derived from un- 
solicited proposals received by the Agency.) This brief state- 
ment might be the beginning of a lengthier cycle of communication 
and interaction focused on collaborative development of proposals 
that reflected the thinking and interests of both the Agency and 
the Researcher (s) . The Agency role here would be largely facili- 
tative. The NIE staffer might suggest that the Researcher get in 
touch with a particular organization that is doing some thinking 
along similar lines, or might suggest that the proposal could be 
strengthened or made more attractive for funding if this or that 
were changed in this or that manner. After several such cycles, 
with perhaps a certain amount of involvement of the broader field 
as well as the j)articular Researcher and staff member, some rather 
exciting and strong plans might evolve, rather different from what 
wc\^ld instead have been proposed without this interactive, col- 
laborative relationship between NIE staff and the Research com- 
munity. We ir^clude this example simply to be illustrative of the 
range of options available for funding Basic Research, options 
rather different in nature from dependence on the unsolicited 
proposal or grants competition mechanisms --with rather different 
implications for the manner in which NIE relates to the field. 

We recognize that the pattern of Agency/Field relationships 
we are offering here for NIE consideration runs counter to 
strongly held feelings in government circles about the need for 
"fairness" — the need to treat all potential contractors and grant 
recipients alike, without giving undue advantage to one or an- 
other individual or organization^ However, we would argue that 
many of these convictions about "fairness" and "equal treatment" 
for potential contractors may be more appropriate to mature 
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systems, and inappropriate to Agency behavior in relation to 
immature R/D§I systems. If NIE is to take seriously its legis- 
lative mandate to "build an effective R§D system" in the edu- 
cation sector, system-building considerations may have to be 
given precedence over other principles of Agency behavior— 
especially when those principles may be inappropriate for the 
particular context in which the Agency must function. We suggest 
that these matters need careful consideration in relation to 
the rather fundamental kinds of questions we are raising about 
NIE's mission and role vis a vis the field with which it works. 



PROBLEM-FOCUSED RESEARCH 

I, GENERIC CHARACTERISTICS OF PROBLEM- FOCUSED RESEARCH 

1 , Introduction 

Problem-Focused Research has two basic characterist ics--and 
herein lies an inherent problem, 

1, Problem- Focused Research is_ Research, 

2, But Problem-Focused Research is "targetted'i--i ,e , , there is 
a very specific area of focus. Thus Problem-Focused 
Research seems like Development, 

From the above, we can see that Problem-Focused Research can be 
deceptive--it looks both like Basic Research and like Development, 
It has elements of both; yet it is neither. Thus, the inherent 
tensions between Basic Research and Development are inherent in the 
Problem- Focused Research process. And this tends to lead predictably 
to certain kinds of problems. 

The basic tension is that: 

1, On the one hand. Researchers tend to treat Problem-Focused 
Research in a Basic Research mode, 

2, On the other hand. Users and Funders tend to treat Problem- 
Focused Research as if it were a Development activity. 

For the funding agency, some important issues arise from this 
basic tension. The tendency of Researchers to use a Basic Research 
mode even while conducting Problem-Focused Research may imply the need 
for some sort of monitoring role from the funding agency. But this 
would be a difficult role for an agency to carry out because of the 
unpredictability involved even in Problem-Focused Research, 

Perhaps even more importantly, the funding agency will have to 
resist its own temptation to treat Problem-Focused Research as though 
it were a Development process. Development permits a greater degree 
of specificity and so of Agency controlling activity. The point here 
is that different--not similar--control mechanisms and strategies 
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are applicable because Development and Problem-Focused Research are 
dissimilar, 

2. Users and Funders: Seeing Problem-Focused Research as a Develop- 
ment Activity 

It is to be expected that Users would tend to see Problem- 
Focused Research as a Development activity. After all, their concern 
is with the development of a product they can use. Perhaps to a 
lesser extent, we would also expect Funders to have a similar per- 
spective, especially given the fact that Researchers often oversell 
their efforts and the potential immediacy of the benefits as a 
strategy to obtain funding. 

Several problems tend to arise when Users and Funders think of 
Problem-Focused Research as a Development activity, 

1, In Development activities, there are specified "targets" 
which are relatively fixed. However, simply because 
Problem-Focused Research is Research, the "targets" tend 
to be unstable. What starts as a Problem-Focused Research 
project ma> gravitate towards a Basic Research project 

(or vice versa) . 

Because Users and Funders have a Development perspective 
(which assumes a stable or fixed "target"), the shifting 
of "targets" in the Problem-Focused Research process easily 
leads to frustration by Users and Funders, 

2, Like any Research activity, Problem-Focused Research has a 
high degree of uncertainty and unpredictability in rela- 
tionship to time horizons, costs, etc. Again, this tends 
to be less so for Development activities , 

When we consider the uncertainty and the instability of 
"targets," we can understand that the RFP-type of mechanism 
is really inappropriate for Problem-Focused Research. The 
RFP mechanism, with its specificity, creates an inevitable 
tension situation, forcing the Researchers to play games. 



59 



The Researchers will tend to oversell, to predict what they 
cannot predict in advance, to suggest they will produce 
something about which they are uncertain. Once having been 
funded, the Researchers are likely to find thty are pro- 
ducing something very different (or that they are working 
in a very different direction) from what the RFP specified* 
Thus, the Researchers face a constant choice between the 
"lesser of two evils": either (a) decline the funding 
because they cannot be sure they can meet the RFP specifi- 
cations; or (b) respond to the RFP and play games with the 
funding agency, 

3. Problem-Focused Research is performed by Researchers who 
behave like Researchers rather than like Development 
personnel. For example, while Development personnel con- 
tin^ially have to face such requirements as specified time- 
lines. Researchers are not accustomed to being subjected 
to such requirements, and will thus tend to ignore and/or 
resent these "Development type" requirements. 

4. In the Need Identification/Problem Definition process, 
there is a real gulf between the Research and Development 
perspectives — and thus between the perspectives of Researchers, 
07.'. the one hand, and Users and Fundars, on the other. 

Users and Funders will tend to focus on the practical 
problem immediately at hand. Their concern will be: 
"l^fhat is going to solve this problem?" Contrarily, the 
User/Funder concern may or may not be seen as important by 
the Researcher, The Researcher will be concerned with 
identifying a researchable problem — which may or may not 
be the same a'c the problem identified by Users or Funders. 

Even whe^i the Researcher and the User/Fvonder are in 
agreement to the desirability of a specific problem- 
focused target, the Researcher may see only one aspect 
of that problem as being researchable- -and the Users and 
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Funders may respond: "Yes, but that won't help us 
to solve our problem"; or, "Yes, but that would take 
too long." 

The issue for the funding agency is the need to balance, on the 
one hand, the significance of the problem area as perceived by Users, 
and on the other hand, the extent to which that problem area is 
researchable, given the current state of the art* The tendency will 
generally be to put too much weight on the significance of the problem 
area and the practical (end-result) justification for beginning a 
Research program, and to give inadequate attention to the extent to 
which the Researcher could likely come up with anything useful at 
this time, or researched in this way, or researched in a certain 
time period. Probably the worst of all situations is to have very 
bad Research being done on a very important problem. 

The nature of Problem-Focused Research is such that the User 
does have a significant role in defining the overall context for the 
Research- -i. e. , in identifying and defining the practical (User) 
need on which the Research is to be focused. However, the User's 
role must be kept in perspective. The User cannot determine what 
is in fact researchable, within what time frame, etc. Thus, the 
funding agency has the role of: 

1. orchestrating the tension between User and Researcher 
perspectives; and 

2. not undertaking program Research in areas that (for 
whatever reasons) cannot be usefully researched. 

5, Problems from the Perspective of the Researcher 

Switching now to the perspective of the Researcher, we find 
that several more problems may be likely to arise because of the 
Researcher's tendency toward a Basic Research rather than a Problem- 
Focused Research mode. 

1. Problem-Focused Research tends to require a scale and 
cost of efforts which is significantly greater than that of the 
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Basic Research mode to which the Researcher is likely to be accustomed. 
This larger scale of effort tends to result from the fact that in 
comparison to Basic Research, Problem- Focused Research: 

a) usually requires significantly more empirical investigation 
(either laboratory or field study); 

b) tends to require more usage of large and possibly inter- 
disciplinary teams. 

What rather naturally tends to happen is that the Researcher 
subsumes or redefines Problem-Focused Research issues into Basic 
Research issues which can be handled within a smaller scale of efforts. 

2. Similarly, the Problem- Focused Research project typically 
needs to be done in some sort of large scale (somewhat bureaucratic) 
institutional complex (such as a Research institution, a Research 
laboratory) where larger scale, longer-term efforts can be made and 
continuity maintained. Again, this is a mode which is not typical 
of Basic Research. 

3. For several reasons, quality control tends to be far more 
of a problem in Problem-Focused Research than in Basic Research. 

a) For one thing, there are fewer and less effective control 
mechanisms. While we can find similar controls in relation 
to Problem-Focused Research, here the controls tend to be 
far more limited in number, scope, visibility and effective- 
ness. Since self quality control of the field is weaker, 
the funding agency will here have to initiate and control 
the use of panels of people in the Research field. A major 
agency concern will be getting the right people together for 
these panels. Where there is a relatively weak Research 
community, the same persons will tend to be over-used, thus 
creating problems of bias and narrowness of focus. Further, 
since Problem-Focused Research tends to be multi-disciplinary, 
various groups can form around specific problems and self 
define the criteria of ^'quality'*, in which case, relevant inter- 
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disciplinary control mechanisms may not exist and may be 
needed* 

When Problem- Focused Research is interdisciplinary, each 
discipline will likely have different perspectives and 
conclusions as to problem definition, criteria for "quality", 
and Research methodology. Thus, the very process of 
coalescing (and orchestrating) the different perspectives 
(and interests) into common agreement is itself a difficult 
process . 

The problem of quality control becomes even more significant 
when viewed from the perspective of scale and magnitude. 
Becaus(5 Problem- Focused Research is generally of a signifi- 
cant lygrea^ter scale or magnitude than Basic Research, 
the need for control in Problem-Focused Research is greater 
than in Basic Research--while the ability to control is 
less. 

At this point, the funding agency faces a very frustrating 
dilemma. The substantial funding investment required by 
Problem- Focused Research calls for increased control of the 
process, but because this is Research, the Agency cannot be 
effective in trying to control the process bureaucratically . 
Thus, the primary means for an agency to control the process 
is to be part of the process--which in turn means having a 
significant internal Problem- Focused Research component. If 
an Agency had such a substantial Problem-Focused Research 
unit (or equivalent), these persons would be part of the 
self-regulating quality control mechanisms of the field. 
Instead of having bureaucratically imposed control, these 
Agency Researchers would be working with the field in • 
creating some standard field mechanisms, the kind of 
mechanisms that provide quality control in the disciplines. 
Because Agency Researchers would be part of the process, 
the Agency would have a degree of control or involvement in 
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the field's own quality control, 
d) We may further note that because of the greater scale and 
cost of Problem-Focused Research as compared to Basic 
Research, funding agencies tend to be more involved in 
the quality control process. Inevitably, this has the 
effect of imposing more limits on the freedom of the 
Researcher. If however, this control were effected in the 
manner described above it might not be perceived as onerous. 

4. The User's Role 

Given the inherent tension between the Researcher and User 
perspectives in Problem-Focused Research, the need is apparent for 
some clarity in defining the appropriate role for the User in the 
Problem- Focused Research process. This would seem to call for 
differentiating appropriate User roles in two stages of the Problem- 
Focused Research process: The Need Identification/Problem Definition/ 
Project Selection stage, and the Research stage, 

1. In the Need Identification/Problem Definition/Project 
Selection stage, mechanisms are needed to allow significant User 
participation because: 

a) The User need is, by definition, a significant parameter 
in determining the focus or "target" of Problem-Focused 
Research. 

b) The Researcher would normally have somewhat different 
insights and perspectives from those of the User. 

Thus, the purpose of User participation in this stage of the 
process is to provide the Researcher with the inputs he needs to 
have a good initial awareness and understanding of User needs from 
the User's perspective . We should be clear, however, that here the 
User role is not one of exercising veto power over project selection. 

2. In the Research stage, however, a rather different User 
role would seem to be called for — one in which User participation is 
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minimal . Here, the premium is on creativeness in response to the 
identified need, and the Researcher is best left alone rather than 
influenced or controlled by possibly dysfunctional User pressures. 

5. Issues Related to the Characteristics of Personnel Involved 
in the Problem-Focused Research Process 

We are now ready to delineate several key issues facing the 
funding agency as a result of characteristics of the Research 
personnel involved in the Problem-Focused Research process. 

A. Large Scale Research and the Response to the Required 
Institutional Setting 

First, we must note that because a considerable amount of 
Problem-Focused Research tends to be large scale (or needs to be so 
carried out) and thus to be performed within an institutionalized 
base, the organizational context may be less than, inviting for the 
Researcher. The extent of btireaucratic control of the Research 
process, and of such things as deadlines and programming of activities, 
is likely to be greater than that to which the Researcher is ac- 
customed. Having been socialized in Basic Research contexts and 
positions. Researchers tend to resist this kind of controlling 
activity. 

As a result, some of the better and more widely known and in- 
fluential Researchers will tend to avoid Problem-Focused Research 
in preference to Basic Research. 

Thus, the funding agency must recognize this dynamic and, if 
anything, bend over backwards to accommodate Researchers in order to 
keep them attracted to the Problem-Focused field (in contrast to what 
agencies sometimes seem to do) . 

B. Redefinition of Studies into a Basic Research Mode 

Even when Researchers* do get involved in Problem-Focused 
Research, they will rather naturally tend to bend the Research to 
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the modes with which they are comfortable, i.e., to redefine Problem- 
Focused Research projects into Basic Research projects. They will 
be more likely to look for the Basic Research aspect of the Problem- 
Focused Research problem than to look at the Problem-Focused Research 
problem at hand. 

Thus, the issue for the funding agency is to keep Researchers 
focused on Problem- Focused Research concerns rather than following 
their natural tendency toward Basic Research questions. The funding 
agency must keep one point clearly and sharply in focus: Problem- 
Focused Research is Research , but it is not Basic Research . Thus, 
a Research proposal should not be funded as if it were Problem-Focused 
Research if it lacks the proper fdcus--i.e., if the proposal focuses 
on Basic Research questions rather than on the kind of problem* 
appropriate to Problem-Focused Research. It might of course be 
worthy of funding as Basic Research — from funds allocated to that. 
Because of the tendency of Researchers to switch the focus, the 
funding agency must be sure it is leading (in consort with the 
Researcher) rather than being led blindly. Its is active and 

proactive instead of passive and reactive. Otheivx^.e funds will tend 
to be directed into Basic Research. 

This is precisely the problem so often encountered in field- 
initiated work (especially work suggested by unsolicited proposals). 
The initiators are often Researchers who, as we are noting, tend 
toward Basic Research, especially the most creative of them and those 
whose efforts tend to be focused on outputs capable of being published 
in the more prestigious Research journals. The potential dangers 
increase when the Researchers making such proposals are the ^'stars'' 



*We are using ''problem*' here to refer to social problems or problems 
of practice, in contrast to the intellectual problems (i.e.. Research 
problems, or problems inherent in the development of the knowledge 
base of a discipline) that structure Basic Research inquiries. 
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of a field. Such persons are often Basic-Research oriented, and 
being influential and prestigious, they tend to become field and 
focus defining, and their proposals are more likely to win approval. 
We must note here that this is not an issue of Excellence. The 
proposed research may be of excellent quality— it just may not be 
appropriately problem- focused. 

The funding agency must mediate the whole process, and this 
will require high quality personnel within the Agency. 

C. Problems in Maintaining the Interdisciplinary Focus 

Similarly, we must note that having been trained in a specific 
discipline, the Researcher will rather naturally tend to redefine 
problems into those that fit into the perspectives and boundaries 
of his own particular discipline— the discipline in which he works 
most comfortably. 

It may be necessary to allow this to happen at least to some 
extent, because this is what is of interest to (and thus motivates) 
the Researcher. The trouble is that it is too easy for the Researcher 
to shift the focus of a problem in this way. 

As a result, where a problem could be studied on an inter- 
disciplinary basis, the interdisciplinary focus may be lost unless 
some kind of proactive tension is provided. 

Thus, the issue for the funding agency is to provide mechanisms 
to maintain an interdisciplinary focus where possible, while at the 
same time being careful not to "overprogram" such an interdisciplinary 
focus. The most reasonable strategy for the funding agency in this 
instance is probably to include this interdisciplinary requirement 
Cs part of the evaluation process at the beginning— during the 
selection of projects, the selection of institutions, etc. --but 
not to interfere with the overall process itself. 

The maintenance of an interdisciplinary focus is an objective 
most appropriately met during the selection stage; it is not likely 
to be accomplished effectively through attempting to impose process 
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controls. In order to maintain an interdisciplinary focus, the 
funding agency has two alternative approaches to consider: 

1. On the one hand, the Agency may choose to create an 
interdisciplinary setting in which the Research will 
be carried out. Here, the Agency must face the issue 
of whether or not it wishes to get into an institution- 
building mode. 

2. On the other hand, the Agency may choose to allow the 
various different disciplines each to undertake the 
work in its own way and within its own boundaries. Here, 
the Agency faces the problem of "putting it all together-- 
which may require additional projects, or the establish- 
ment of some other method of integration. 

6 , Summary 

Inherent in the Problem-Focused Research process are two 
conflicting tendencies; 

1. The tendency of Users and Funders to turn it into a 
Development process. 

2. The tendency of Researchers to turn it into a Basic 
Research process. 

A further complicating problem is the fact that the Researchers are 
not a cohesive group but rather belong to different disciplines, each 
with its own perspectives and boundaries. 

The funding agency must be very clear as to the existence and 
nature of these conflicting forces. 

In this light, the funding agency has several key roles: 

1. to maintain the integrity of the Problem-Focused Research; 

2. to mediate, balance, synthesize the conflicting forces 
inherent in the Problem-Focused Research process; 

3. to insure that there is adequate input regarding the per- 
spectives and needs of the various kinds of Users to which 
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the Research process is directed ; 
4. to insure the participation of competent Researchers by 
presenting clearly and invitingly the Research aspects 
of the process which interest, excite and motivate them. 
.It should be clear by now that balancing the conflicting forces 
in a way that keeps Problem-Focused Research productive requires 
skillful orchestration by the funding agency personnel. It is a 'so 
immediately obvious that if the funding agency is to be able to 
perform its orchestrating role effectively, it would seem advisable 
for the Agency to have very knowledgeable and skillful personnel on 
its staff— personnel who understand the forces and issues involved. 
As this is not always possible, an alternative that might be attempted 
would be to utilize such integrating mechanisms as panels, conferences, 
etc., where representatives of the Users and of the various disciplines 
of the Researchers can discuss the issues and reach consensus. How- 
ever, we must recognize that the basic nature of the Problem-Focused 
Research process is for the different participants to be in tension 
with one another. Thus, just bringing them together does not solve 
the problem of conflict— it merely provides a way or mechanism for 
orchestrating a conflict sitution. As a result, even when provision 
is made for bringing the different participants together, there 
remains the need for in-house Agency people to perform skillfully 
the role of balancing, mediating, and orchestrating the conflicting 
forces . 

In the way the funding agency attempts to procure programs ar/: 
manage Research activities, there is really no substitute for having 
within the Agency knowledgeable and skilled personnel who are highly 
involved in the area of concern. 

The generic characteristics of the Problem-Focused Research 
process are so important that we f-obably cannot overemphasize what 
Problem- Focused Research is and is not— and especially how it differs . 
from Basic Research and Development. 
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From this perspective, we have already noted the following 
implications for Agency strategy: 

1. The process cannot be controlled in a standard "bureaucratic" 
manner. Rather, an agency can best exercise control by 
being involved in the process. 

2. The Agency must piay a major leadership role rather than 
being blindly led by the Researchers. At the same time, 
Problem-Focused Research is not highly attractive to some 
of the most creative Researchers. Thus, while leading and 
controlling, the Agency 'must also be as accommodating as 
possible to the Researchers in those aspects that do not 
subvert or invert the primary intent of the program. 

3. It follows that having knowledgeable professionals within 
the Agency is a very important part of a funding strategy. 

II. THE EDUCATIONAL CONTEXT OF PROBLEM- FOCUSED RESEARCH 

1. Basic Weaknesses 

' Probably the greater part of the Research that is carried on 
within the field of education is Research of the Problem-Focused type. 
There is a sizeable community of educational Researchers who carry 
out work that is problem- focused. And there is an enormous quantity 
of Research of this applied or problem-focused variety produced every 
year. Nonetheless, the field is basically a weak field: 

1. Most of what is happening is small-scale, scattered, 
fragmented. 

There are major questions about the quality of what is 
being done. 

^. There is a lack of clear definition and identity of the 
field as somehow different from either Baoic Research or 
Evaluation or Policy Research. 

4. There is even a lack of clear delineation or consensus on 
the problem areas that structure this as a field of inquiry 
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or as an R/DSI function. 

There is a considerable amount of confusion in education as to 
just what is and is not Research of an Applied or Problem-Focused 
nature. On one end of the inquiry continuum, one finds considerable 
difficult in education in distinguishing Applied or Problem-Focused 
Reearch from Basic Research. On the other end of the continuum, 
there is a tendency of many- -generally non-Researchers--to confuse 
Applied or Problem- Focused Research with a broad range of other 
activities that cannot properly be subsumed under the Research rubric — 
e.g., library research (such as examination of alternative sets of 
teaching materials); demonstration projects (a favorite device for 
diverting "Research" funds into other uses); or the social bookkeeping 
kind of statistical record keeping that occupies the attention of so 
many school system "Research" offices. 

One of the clearest indicators of the weaknesses and immaturity 
of the Problem-Focused Research field is the widespread inability of 
educational Researchers to define what is a researchable problem. In 
education, a great deal of Hvhat is defined as a problem is defined on 
the basis of the significance of the topic area rather than on the 
basis of what is researchable. Indeed, many of the proposals that are 
made are about questions having clear significance in terms of the 
substance of the problem. However, the simple fact that a significant 
social or practice-based problem is not itself a Research problem is 
something that does not appear to be completely* understood in the 
field of education. The oft-noted criticism of the inability of 
educational Researchers to define reasonable Research problems comes 
about in part because there is_ a gap between Researchers' understanding 
of a given need or topic and how one would go about researching the 
need or even translating the general problem into a researchable 
problem. As a result. Problem- Focused Research in education is 
characterized by a tremendous amount of oversell. This is not because 
the people involved are charlatans, deliberately promising to answer 
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questions they know they cannot answer. Rather, it is too often the 
case that the Researchers do not even know that the state of the art 
renders them incapable of answering questions which they have defined 
as a need or problem. 

2 . Institutional Bases 

There are primarily two institutional bases where Problem- 
Focused Research is carried out in the field of education: the 
universities, and a number of large R^D organizations in the private 
and quasi-public sectors. There are problems inherent in both of 
these institutional bases. 

A, The Universities 

The universities have the capability to do Problem-Focused 
Research, but they tend to do it in a Basic Research mode--both 
because that is the mode the Researchers have been trained and 
socialized in, and because of the social pressures of the university 
as an institution. In a university setting, Researchers are rewarded 
for acting like Basic Researchers rather than for problem solving. 
Thus, Research gets bent into the Basic Research mode, especially 
if the Research is being done by a group within a particular discipline 
A second point about universities is that they usually do not have 
a minimum critical mass of the capabilities needed to carry out 
Problem- Focused Research. The result tends to be that either (a) 
a problem that requires a large scale approach will tend to be scaled 
down to fit the resources that the university has available; or (b) 
as an alternative, the university will put together an ad-hoc group 
of people who tend not to stay together very long as a work group. 
Neither of these alternatives is really very helpful, but both are 
typical enough to warrant some further attention. 

1 . Scaling Down the Problem 

If a university lacks adequate resources to deal with a 

problem that requires a large scale approach, the problem will 
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tend to be redefined to a smaller, manageable scale. In 
particular, the time frame will tend to be reduced (as we 
shall note in more detail below). Additionally, in a uni- 
versity setting a problem is often approached by a group 
of Researchers from a specific discipline and, rather natur- 
ally, this group will scale the problem down to fit into the 
focus and capabilities of their own disciplines, tl^ veering 
the Research away from the problem focus to a disciplinary 
focus. In schools of education, where the problem focus is 
more likely to be maintained, a somewhat different kind of 
problem is encountered frequently. With the possible ex- 
ception of the ten to fifteen schools of education in this 
country with strong Research traditions, schools of education 
generally lack the kind of well trained Research faculty and 
graduate students more likely to be found within the dis- 
ciplinary departments. Typically, then, the university setting 
for Problem-Focused Research offers the funding agency a choice 
between well trained personnel who are likely to redefine Problem 
Focused Research into Basic Research modes or adherence to the 
problem focus by Researchers less likely to produce quality work. 

2. The Ad-Hoc Approach 

Whether the reason be the disciplinary emphases of university 
personne,!^ the general fragmentation of the field, or whatever, 
it is the nature of the. , university social system that people 
will tend to come together for a while to work on a problem 
and then go their separate ways instead of staying together 
for a long period of time. 

This ad-hoc, short term approach tends to be highly dys- 
functional simply because Problem-Focused Research (and 
Research in general) tends to require longevity— i .e . , the 
building of skill groups that can work together over long 
periods of time. The focus of the Research is problem solving. 



a process that takes time, e.-pecially if the ne^^c' is to 
develop a long-r^ng^/ Probiein-Focused ReiTearch program* 

We may alr;0 note that the commitment to a set or series 
of concern: (which tends y:o exist in Rasic Research in the 
disciplinary setting) tent'.^ more often than not to be missing 
in relation to Problem-Focused Research--a dynamic that re- 
i-nforces the ad-hoc approach. 

This tendency towards ad-hoc appioaches bears further 
examination from the institution-building perspective. There 
may indeed b& extremely well written proposals from a uni- 
versity group to do ?,esearch in a given area. There may even 
be a ^^star'', a very comp^twt Researcher with the appearance 
of a strong group around him Co^ her) to do the Research* 
Indeed, for providing good quality resc/JltS from a particular 
piece of Research, this group may in fact app^?ar to be a strongly 
qualified group, one that would do the work well. 

However, it would seem essential for NIE to be concerned 
with long-term capacity and system building as well as with the 
particular results of a single Research project* Looked at 
from an overall system perspective, the procurement of a group 
that will be together only for one or maybe two projects will 
not be as relevant, meaningful or cost/effective (in the long 
term) as funding a group which has (or has the potential to 
develop) longevity* In the latter instance, the Agency would 
be procuring not only a short-term product but also a system- 
building capacity* 

In summary, universities tend not to be able to put together 
long-term, large scale work groups* As a result they tend either 
to scale down a problem into one that is manageable with existing 
capabilities; or they develop ad-hoc, short-life groups for 
individual Research projects which add nothing of significance 
towards building overall system capabilities. 
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B. Large RSD Organizations in the Private and Quasi-Public 
Sectors 

A somewhat different set of problems characterize Problem- 
Focused Research as this is carried out in large R§D organizations 
in the private sector (generally non-profit corporations and oc- 
casionally for-profit organizations) and in the quasi-public sector 
(the federally-created regional laboratories and R§D centers). These 
are organizations that, by their very nature, should be particularly 
well-suited to carrying out Problem-Focused Research. They are 
likely to develop and put together the resources for large scale 
Problem- Focused Research, particularly the larger organizations. They 
are likely to put together work teams that will stay together for a 
long time. There are not the internal social pressure that one gets 
in the university setting that would veer Researchers away from 
focusing on specified problems (even when these seem to provide no 
exciting and potentially publishable outputs) . 

However, at least two significant kinds of problems are dis- 
cernible from the history of these organization. First, it is not 
obvious that there are attractive career paths in these organizations 
that are capable of attracting the best talent. A small handful of 
these organizations win a large number of contracts, and can offer 
people some degree of stability and the prospect of a long-term career. 

compared to a university (which is the setting that tends to 
attrac'^ most of the best Research talent), there is no similar kind 
of long-term tenure or coramitm.ont. The likelihood is that, for a 
variety of reasons, these organizations may have great difficulty 
attracting the strongest Research talents. 

The second kind of problem may be an even more serious one. 
Many of these organizations were originally established to carry out 
Applied (i.e., Problem-Focused) Research as well as Development. 
However, the history of the labs and centers is suggestive of a 
pattern that is probably somewhat discernible in the private sector 
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as well. By the later 60s, federal procurement emphases reshaped 
these organizations, especially the labs, into Development organiza- 
tions rather than organizations that carried out the full spectrum 
of R/D&I activities including Problem-Focused Research. That is 
to say, most of the money tended to go for Development rather than 
Problem-Focused Research. Consequently, these organizations (being 
the type that are structurally responsive to funding sources) ex- 
panded their Development capacity more and more. To the extent they 
carried on Problem-Focused Research, it might have been as an induce- 
ment to keep some of their Research talent happy and to maintain them 
in the organizarion. 

To summarize, there is one all-pervasive point that becomes 
apparent from examining the institutional bases of Problem-Focused 
Research in education: there is a fair amount of Problem-Focused 
Research around, but it tends to be discrete, small scale Research 
projects rather than very large-scale projects. Certainly there may 
be a significant number of Problem-Focused Research problems which can 
be tackled in a small way. But the ones that are probably the focus 
of NIE procurement tend to be larger-scale, complex programs that 
will likely require long-term Research capability. 

The basic point is that there is very little large-scale, long- 
rerm Problem- Focused Research being done. The universities tend to 
scale it down or not do it in the long-term mode. The large insti- 
tutes and organizations in the private and quasi-public sectors that 
are organizationally capable of putting together large scale long-term 
work teams tend to have been pushed into becoming primarily Development 
organizations. This may be overstating the case; but certainly they 
have not filled the need for a significant amount of largo-scale, 
long-term Problem-Focused Research to meet needs. 

It appears that Problem-Focused Research in education is 
essentially an area that needs to be put together--it it not really 
there now to any sizeable degree. In part this is because the field 
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has tended to think in terms of Basic Research concepts^ or Develop- 
ment concepts, rather than thinking of Problem-Focused Research as 
a fundamentally different kind of Research, requiring different 
sorts of structures and institutional bases which are suited to 
Problem-Focused Research as a distinct mode of R&D activity. 

5. Some Additional Considerations 

A. Communication 

Problem-Focused Research shares many of the problems that we 
already noted about Basic Research. The information mechanisms and 
the communication structures of the field are inadequate--even more 
so than in Basic Research. Not only does the field lack invisible 
col leges, but there is an even weaker journal structure. Relatively 
few journals specialize in definable Problem- Focused Research areas, 
and material relevant to any particular problem area tends to be 
scattered among scores of journals. There are relatively few 
abstracting mechanisms in the educational field at all, and virtually 
no usable abstracting mechanism of relevance to any but a handful of 
problem areas. The whole information retrieval system, which might 
be potentially useful to the Problem-Focused Researcher, is really 
not geared to Problem- Focused Research. 

B. Commitment 

Given the he- field is currently organized, with the Problem 

Focused Researchers constantly moving in an out of problem areas, 
commitments tend to be short-term rather than long-term. Thus in-' 
formal communication mechanisms and personal information flows are 
difficult to sustain; the communication networks are even more 
diffuse than in Basic Research, where we have already noted the very 
rudimentary state of development of communications networks. 

C. Climate 

The climate. surrounding Problem-Focused Research is even more 
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negative than tiie climate surrounding Basic Research. The reason, 
quite simply, is that more tends to be expected of Problem-Focused 
Research, and therefore the disappointment is far greater when no 
payoff seems forthcoming. Basic Research does not hold out the 
promise of solving life's problems so saliently. Thus, there is a 
degree of acceptance when a specific Basic Research project does not 
produce answers--or may do so only in a distant future. 

But, in contrast, precisely because it is problem focused , 
Problem-Focused Research does (rightly or wrongly) seem to hold out 
a "promise" to solve ongoing operational problems. Thus, when such 
rather costly Research projects do not produce answers (or might 
possibly do so only over a long period of time), there is_ disappoint- 
ment. The result, inevitably, tends to be the generation of strongly 
held feelings that Problem-Focused Research is a waste of money. 

D. Need Identification 

A related point here has to do with Need Identification for 
Research--who does it, and how a.whole'set of problem areas have 
been identified. In the past. Need Identification has been 
Researcher-driven. That is to say. Researchers identified their own 
problems, with very little input from Users. Thus, the problems 
defined by the Researchers were things that interested them rather 
than necessarily the problems perceived by Users or by people 
affected by the problem. As a result. Researcher-defined problems 
turned out to be either so irrelevant, ill-defined or misperceived 
that the User community perceived the Research as useless. 

Now it appears that there has been an overreaction to the pre- 
dominance of Researcher-initiated inquiry into problems. Thus, the 
definition of problems is now perhaps overly system-driven. NIE is 
defining problems; Users are defining problems; and the Research 
community is doing a great deal of complaining about the fact that 
the problems which are being defined either are not problems that are 
researchable or are not problems that should be the focus of Research 
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sponsorship. 

Clearly, then, there is some need for orchestration in the Need 
Identification process. Given the current level of maturi"*:/ in 
educational R/D§I as a whole and educational Problem-Focused Research 
in particular, only an agency such as NIE can provide the orchestra- 
tion needed to maintain and mediate the existing tension, permitting 
the Researcher viewpoint to take on increasing influence as the 
capability of the Research community permits. 

E . Funding 

As in Basic Research, the sources for Problem-Focused Research 
funding are numerous. NIE, as the lead agency (even though not 
necessarily the major funding source), has a role in working with other 
agencies to orchestrate funding so that NIE funds can be applied in a 
way maximally useful to the system. ^ 

III. Implications for the Role of NIE 

1 , Procurement Behavior 

A. Orchestration 

Given the basic tension inherent in Problem-Focused Research 
and the resultant tendencies for Researchers to treat it as 
though it were Basic Research while Funders and Users tend to 
treat it like Development, NIE has a key orchestration role in 
mediating the inevitable conflicts between Researchers on the 
one hand and Users and Funders on the other hand- -as well as 
reconciling differences in perspectives among Researchers from 
the various disciplines relevant to a given problem area. 

Critical here is the orchestration of User and Researcher 
perspectives in problem selection--balancing Users' interest in 
targetting Research at problems where they perceive a real need 
and Researchers' concerns about the researchability of identified 
problem areas given the state of the art in relevant fields. 
If high quality Researchers are to be attracted to and maintained 
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in large scale educational Research institutions that carry out 
Problem-Focused Research, the field and the institutional settings 
must be made attractive to them. As far as is feasible the forms 
and modes of funding and control must be made to accommodate their 
needs. It is vital to include the User perspective of problem 
area significance in initial project and program selection, but 
the overall balance will need to be redressed in favor of the 
protection of the Researchers' criteria of Excellence and re- 
searchability (so often lacking in the current situation). 

This kind of orchestration will require great sensitivity 
and skill, and suggests the need for Agency personnel: a) who 
can work closely and well with both Researchers and Users; 
b) who can gain their trust and confidence while still maintain- 
ing the integrity of the Institute's perspective; c) who can 
intervene appropriately to make certain that the Research 
maintains its problem focus and that the problems selected for 
study both meet User needs and are in fact researchable and 
carried out in a manner likely to be productive. 

The use of advisory panels to help NIE staff define Research 
program areas would seem appropriate. But their influence may well 
be somewhat limited until the field gains in strength, A danger 
to be avoided is the filling out of these panels with dispro- 
portionate representation from the Basic Research community (no 
matter how distinguished these Basic Researchers may be, especially 
those from the disciplines) and from the User community. Until 
more can be delegated to a stronger interdisciplinary field, NIE 
will have to be prepared to assume a larger share of the overall 
programming responsibility, 

B. System Building 

The fundamental point we emphasized in our analysis of the 
educational context was the relative absence of Problem-Focused 
Research settings with the required attributes of large scale, 
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longevity, and Excellence. Further, we noted that prior funding 
policies had tended to transform the centers of applied work 
that do^ exist into what are primarily Development organizations. 

The implication for NIE is that institution building (and 
rebuilding) is a very important need--and only NIE (through its 
funding, orchestrating, and managing) can provide the impetus 
for such a strategy. 

Institution building implies: 

1. locating existing or potential v^enters of excellence; 

2. developing and supporting them in a manner that will 
permit the longevity of work teams and the kind of 
scal<^d operations that Problem-Focused Research often 
requires , 

Specifically in relation to universities (as one obvious 
source for centers of excellence), the problem is the tension 
between (a) the need to keep the focus on Problem- Focused Research; 
and (b) the fact that Problem-Focused Research is generally not 
rewarded in the university setting. Thus, two alternative strate- 
gies could be: 

1. to attract competent university Researchers to R§D 
organizations in the private and quasi-rpublic sectors 
(which, however, would have the dysfunctional effect 

of weakening potential university centers of excellence); 
or 

2. to provide joint-appointment arrangements wherein 
university Researchers can both maintain their univer- 
sity relationships (and security) yet also devote major 
amounts of their time to Problem-Focused Research in 
these RSD organizations outside the university setting. 

In relation to large-scale R5D organizations in the private nnr^ 
quasi-public sectors, NIE's strategy will probably have to be one 
of re-building--i .e. , developing funding policies with the needed 
focus, scale, and time-frame to permit these organizations to 
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return to the Problem-Focused mold. 

As in our discussion of institution building for the Basic 
Research function, system, building will require a somewhat close, 
collaborative relationship with the field to permit the identi- 
fication of potential 'Zenters of excellence with the required 
"minimum critical mass of talent." And as in Basic Research, 
with limited resources for funding and a wide array of potential 
problem areas for the focus of such funding, a policy of "placing 
bets" and concentrating funding on a few selected problem areas 
would seem preferable to scattering the available resources across 
a much larger range of problem areas and Research organizations. 

The funding of Problem-Focused Research in education has been 
characterized for some time by a Catch-22 problem i.e., a funding 
pattern which requires Researchers to promise unattainable (or at 
the very least, highly uncertain) results in order to obtain 
funding. Thus, if you promise, you can't deliver. If you don't 
promise, you don't get funded. 

NIE--and Users, Researchers, and the Congress--need to clearly 

understand that: 

1. The problems selected must be important problems. 

2. At the same time, the problems must be researchable . 
5. The capacity does not really exist right now to 

meet those needs. 

4. Thus, institution building (anc rebuilding) is the 
prime need. 

5. However, institution building is an ongoing, long-term 
process . 

These points imply that Problem-Kocused Research must be 
promoted as an iipportant but long term contributor to education-- 
one that requires a major investment now if the gap is to be 
filled, more for future than for present benefits. Perhaps this is 
the only way to overcome the Catch- 22 problem. Vital to such a 
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strategy is that Users, the Congress and NIE have a clear under- 
standing of the differentiation between Problem-Focused Research 
and Development. 

C. Process Mode of Management 

Given the size of the investment required for large-scale 
Problem-Focused Research, and recognition of the somewhat limited 
state of development of the field, a funding agency is like!/ to 
have a strongly felt need to exercise some control over its Re- 
search program. But, as we have emphasized repeatedly, Problem- 
Focused Research i£ Research, and it is carried out by Researchers. 
Therefore, it would seem essential for NIE to avoid a bureau- 
cratic response. Tightly drawn RFP's that attempt to mandate 
targets, deadlines, procedures, etc. have limited likelihood 
of being effective and seem to be inappropriate substitutes for 
knowledgeable close interactive relationships between NIE staff 
and the field, especially where NIE staff include Researchers 
who function as an integral part of the field. 

Consi derations Internal to NIE and Non-Procurement Activitie s 
A, Staffing 

Recognizing the high cost of establishing large scale Problem- 
Focused efforts of appropriate critical mas^s, the visibility of 
such efforts, and the tremendous difficulty of keeping such work 
problem- focused (especially during a period in"^ which the emphasis 
is io be on institution building and on attracting strong Research 
talent), NIE will be faced with the need to build a very strong 
internal group capable of close working relationships with the 
field--relationships that imply a substantial degree of leadership. 
This can be achieved only if such personnel are themselves involved 
in impo-i:ant Problem-Focused Research and have achieved some stature 
in the Research community. 

Without such a relationship and a strengthened field (backed 
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up by the needed in-house competence and activity), merely doubling 
or tripling the proportion of funding allocated to field-initiated 
Research would not seem to be a purpo- " strs agy in and of it- 
self. Unless NIE is capable of orch a- ng the Basic Research/ 
Development tension, the quality con jquirements, the 

institution building programs and the much greater amount of fund- 
ing avail;.ib7. ^ ^ - Problem-Focused Research from other agtacies 
and, furth-.\ capable of performing the monitoring of the total 

effort, then --^c ly turning more of the initiative over to an 
overtly weak field would seem to be futile. 

B. internal Agency Structure 

Given what we have said of the importance of separating 
Problem-Focused Research from both Basic Research and Development, 
it might be desirable for this separation also to be reflected 
in both organizational and budgetary structures within NIE. This 
would seem to be a matter that merits some attention within the 
Agency . 

C. Non-Procurement Activities 

Much of what we discussed earlier in relation to non-procurement 
activities of the Agency appropriate for the Basic Research func- 
tion applies equally well to Problem-Focused Research. Included 
here mignt be such activities as: NIE Researchers conducting 
Problem-Focused Research of their own, possibly in collaboration 
with Researchers in the field; keeping in touch with the field 
through travel, telephone communication, attending and/or sponsoring 
seminars and conferences, etc.; facilitating the development of 
invisible colleges and other communication and qualit* control 
mechanisms needed for the cumulative development of relevant 
knowledge and technology bases; etc. 

Here too, NIE Researchers might nap potential problem areas, 
prepare syntheses and reviews of the state of available knowledge, 
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etc., apd use such documents for recruiting and attracting top 
Research talent to specific Problem-Focused Research areas — 
in much the same manner described earlier in our dis'^ussion of 
Basic Research. 

The procurement implications of these kinds of staffing and 
non-procurement activities parallel those considered earlier 
for Basic Research. Here too, close interactive relationships 
between Agency and field, and collaborative development of 
Research proposals and programs, would seem possible if system- 
building considerations could be seen to take precedence over 
the rules of the game of competitive procurements. Inherent in 
this discussion may well be one of the most significant and 
controversial policy issues in need of NIE consideration — one 
fraught with enormous implications for the Institute as well 
as for the whole educational R/D§I system and its future develop- 
ment . 
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DEVELOPNffiNT 



I, THE NATURE OF DEVELOPMENT 

1 . Introduction 

A. User Focus 

Development involves a process of converting knowledge into 
User-ready products.* The emphasis or focus of Development is the User. 
The end-product must be something a User can use (wxth at most some 
minimal fitting or tailoring). Thus, we use the term "User -ready*' to 
emphasize that when a pr luct is "developed," it is in a form that the 
User can potentially use — it is "ready" for use. 

It follows rather obviously that Development is very dependent 
upon and thus must be very responsive to User needs. Further, we must 
emphasize that Development must be related to current User needs — not 
to some "foreseen" long-term future needs. 

Of course, there is always the possibility to foresee (or project) 
a time when Users will be ready for a product which they do not currently 
see as a need. However, this kind of situation tends to be much more 
theoretical than real. Normally, a Developer h-is neither the capability 
to make such "putting products on the shelf" projections accurately nor 
the luxury to do so. 

B. Dependence upon the Development State of the Art 

Whether or not it is even possible to develop a specific product 
is, of course; determined by what the state of the art permits. Whether 
or not a product should be developed also depends on the state of the 

*As noted in an earlier chapter, for simplicity of usage we use the term 
"products" whenever we are referring to the outputs of the Development 
process. However, the reader should keep in mind that the term "products" 
is meant here to convey the full array of Development outputs- -programs, 
processes, models, st-'rt^gies, approaches, etc. as well as the narrower 
range of outputs we typically think of as "products." 
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art. For example, there is no point in creating a product that is in 
fact out-of-date if the state of the art permits a superior product to 
be developed. 

We must note that we refer to the Development state of the art -- 
not to the Research state of the art. As we will note later, there is 
in fact very little direct transfer from Research into Development. The 
products of Research appear in long-term, indirect ways. By contrast. 
Development products generally come from a more certain, established 
knowledge base i.e.. Development products tend to be based upon or 
are an extension of other kinds of Development products. In simple 
terms, an existing product is modified, improved, etc., or a well 
developed concept is turned into a usable product. 

C. Development anc Production 

Development is the creation of a product in a form in which it 
can be used directly or duplicated for wider use. 

In the gen.ric sense, the end of Development is the beginning of 
Production. Once a product is developed, Production begins a.id at some 
later time (usually soon after Production begins), the Producer dis- 
seminates (distributes, markets) the product to the User. 

In some fields (where the products are essentially technologies, 
e.g. i.t education), this clear-cut process of Development/Production/ 
Dissemination/Utilization generally does not exist. In such cases, the 
actual Production may be carried out by the Users themselves. After 
Development is completed, the next step is usually simply informing 
the User how to reproduce the developed product. Thus, in education, 
a clear-cut, separate Production stage often does not exist. 

D. The Steps in Development 

The Development process will depend to some degree on the nature 
of the product being developed. However, in one form or another the 
process w' 11 generally Mnvolve: Need Identification, Development justi- 
fication and feasibility, design, model or prototype building, testing, 
packaging (which may include creation of Implementation/Utilization 
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instructions), pilot dissemination, large-scale field testing, final 
modifications. 

2. The Centrality of System Linkages 
A. T^es of Linkages 

In Development, three system linkages are of critical importance; 
i.e., the linkages betv^een the Developer and: 

1. the User (for Need Identification) 

2. the Development state of the art 

3. the Production stage. 

1 . Need Identification Linkages with the User 
Normally we tend to think of Need Identification as occurring 
at the be ginning of the Development process--i . e . , wh^n the User 
(either directly or through some l.itermediary) works w^th the Developer 
to define the area of need. Typically, Need Identification is then 
considered to have been completed. - 

This, however, is a very limited concept, one that works well 
only under conditions of overall system maturity and certainty--i . e . , 
when : 

1. thie Users can clearly specify what they need; 

2. the Developer (and Producer) can know exactly what the 
Users mean; 

3. the Developer is then capable of producing the developed 
product and theji sa/ing with assuredness to the Users: 
»'Here is what you asked for'*; 

4. it is then obvious to the Users what to do with the product. 
As an example, an airplane manufacturer may well be able to specify so 
clearly the requirements for a needed airplane part that the part can 

be developed to specifications and then, in effect, simply "plugged in.*' 

In contrast, of what valun is <^.uch an isolated, simplistic 
process of Need Identification Uu^^^ conditions-of overall system 
immaturity and uncertainty (such as in education), when the User 
cannot clearly tell th Developer what is ne<^dcu, when the Developer 
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would not really be sure how to go about developing the product even 
if the User were to give clear specifications, etc.? 

Under these conditions, Need Identification must be seen as 
a broad, ongoing, continuous process — a process that: 

1. enables the Developer throughout the various stages of 
Development to seek and receive additional information 
and clarification from the User; 

2. not only involves successive stages of the Development 
process, but is continuous through Dissemination, 
Implementation, and Utilization, until the need has been 
refined sufficiently for a usable product to be developed. 

In a word, under conditions overall system immaturity and un- 
certainty. Development is a continuous process of adjustment and 
modification--a tailoring of the product which does not end With the 
generic '^Development*' stage but continues through the User aspects 
of the R/D&I process. 

2. Linka ges to the Development State of the / ct 

While not as critical or as difficult as linkages between the 
Developer and the User, the linkage between the Developer and the 
Development state of the art is nonetheless important. The Developer 
does need to know what has been done, can be modified or improved, 
can be applied to a particular project. 

The linkage is to the Development knowledge base. This does 
not mean that linkage to Problem-Focused Research cannot at times be 
helpful. (Indeed, some Problem-Focused Research quite often occurs 
concurrently with the Development process- -i .e. , as the Developer 
runs into ^'problems . This also means that the linkage to Basic 
Research is at best tangential and occasional. 

5 . Linkages to the Production Stage 

A not uncommon breakdown in the R/D&I process occurs when a 
deve loped product is given to the Producer in a form that (for whatever 
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reason) is not practical or feasible for Production purposes. This is 
not surprising. If Developers are doing their job, they have kept in 
mind that the final product must be in a form which the User can use. 
We simply note here that the Developers also need to consider that the 
developed product (prototype, model) can be manufactured , where such a 
function is needed. To know what are the Production constraints, 
criteria, etc., the Developer needs communication linkages with the 
Producer. 

B, The Agency Role: Monitoring and Orchestrating System Linkages 

The mission of a funding agency in Development is in essence to 
insure that adequate Development is (or can) occur. Because Development 
is a system phenomenon, x major Agency responsibility is to monitor and 
orchestrate the system. l.\nj{ages. We emphasize the system nature of 
Development simply because Development cannot be isolated from 
Production, Dissemination, Implementation, and Utilization, 

In a mature system, this role is likely to be relatively smaller, 
as there may have developed natural linkages which provide for a high 
degree of self-monitoring and self-orchestration, 

Contrarily, in the uncertain conditions of an immature system, 
monitoring and orchestration may be a significant Agency role. 

THE EDUCATIONAL CONTEXT OF DEVELOPMENT 

1 . The L ack of a ProJuction Fuiiction in Education 

As we have already noted briefly, in the educational context, 
Development often does take on aspects of Production— simply because 
a separate Production stage and apparatus often doe.*5 not exist. 

When no mass production is involved, the developed prototype may 
indeed be the sum total of Prodraction, Then the issue is simply one 
of informing Csers of the prototype. 

In other instances, the Production phase may occur at several 
Tin olated points in time (instead of over a single, continuous point 
n cime). The prototype or model is put into the system, in the 
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literature, in ERIC--for retrieval by Users whenever they wish. 

Of course, there are types of outputs (games, books) for which 
there may be a distinct Production stage. Here the issue to be con- 
sidered is: who is willing and capable to do Production? 

In cases where the Development work has been done by a commercial 
firm, the carrying out of Production activities is usually not a 
problem. Commercial firms tend to have well developed Production 
capabilities. And their right to control the nature of the output in 
its final form cannot really be challenged given their joint role as 
both Developer and Producer. 

However, when the Developer is government-funded, questions about 
commercialization in the Production stage arise inevitably. Production 
considerations may suggest major changes in the Developer's prototype 
if Production is to be carried out on a sufficient scale and/or if it 
is to be kept within specified costs. In such instances, the Developer 
will have to re oncile his desire for Production on a scale that would 
permit widespread Dissemination and Utilization, with his desire to 
maintain the integrity of his prototype and to control the character of 
the final product. This is clearly an issue worthy of some attention, 
but one that we cannot deal with here. 

Mention should also be made of the difficulty encountered occa- 
sionally when commercial Producers acquire the rights to a larger 
number of Development outputs than they can readily produce over the 
short run. The consequence here may be that the product's dissemination 
is delayed for some time (if not, perhaps, forever). Tliis potential 
problem suggests the need for considering the possibility of agreements 
with commercia Producers thac entail forfeit of the Producer's rights 
to a product if its Production and at least the beginnings of its Dis- 
semination are not accomplished within a specified time period. 

For a variety of reasons, a product may not be commercialized-- 
either by intent or by default, or for lack of interest from commercial 
Producers. Where Production activities of some kind or other are called 
for by the very nature? of the product, then the question becomes: if 
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not commercial Producers, who will do Production? There are a number 
of possible answers. Perhaps the Developer will do Production per- 
haps even Lixe User or various kinds o£ intermediary organizations created 
to link Producers and Userr,- The character of that Production may well 
vary depending on who in tiie system actually carries out the Production 
work. 

2. T he Development Knowledge/Technology Base 

There seems to be general agreement that the educational Develop- 
ment knowledge/technology base is weak, that its quality is generally 
poor. Even when one can point to some highly valuable Development 
knowledge or some highly useful Development technology, access to 
quality information of this kind is difficult: 

a. Developers do not know what other Developers have done or 
can do. 

b. There is little codification of Development knowledge/ 
technology-- the kind of codification to which Developers 
could turn and on which they could rely. There is an 
absence of "handbooks.*' Lacking too are distinct, 
discrete. Development-relevant categories to facilitate 
information searches . 

c. The transforms or .linkage mechanisms be ./een stages of 
the educational R/D§I process are weak (e.g., there are 
so many unknowns in education that Research findings do 

^ not readily get translated into Development products; 

almost equally arabigious may be the linkages between 
Development and Production or Dissemination) . fonsidering 
that Development is a continuous process acroHS various 
R/D5I stages, such transforms or linkages are quite 
important . 

In light of the above, quality control becomes a key issue for 
educational Development. Further, since the quality control needs to 
be continuous throughout a.id even beyond the Development stage, there 
is a strong logic for' breaking up the procurement of Development into 
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stages. As an example: 

Stage 1; The Developer is asked to present NIE with a concept 

for development of a product. 
Stage 2: The Developer is asked to create and pilot test a 

prototype . 

Stage 5; The Developer is asked to develop a scaled up version 
ready for large-scale Dissemination. 

At each stage, the Development process could be reviewed, re- 
fined and e .i stopped. The point is simply that NIE cannot afford to 
assume that adequate quality control will be provided throughout an 
entire Development cycle. 

This stage-by-stage quality control could be accomplished through 
either of two basic strategies. 

1. A single Developer could be funded for the entire Development 
process, but NIE would review the Developer's work at various 
stages in the process. 

2. Each stage of the process could be funded separately. This 
would allow NIE the option (at each stage) to continue with 
a single Developer or to open each stage to competition. 

The point to be emphasized here is simply that NIE should think 
about Development prccurement along lines such as these. And if NIE 
is going to consider such strategies, a fairly active NIE involvement 
is required — as contrasted, for example, with NIE simply providing 
long-term funding to a Developer who proposed what appears to be an 
exciting, potentially significant Development project. Such NIE in-r 
volveiTient may seem onerous at times--but in educational Development, 
some active degree of NIE involvement is bound to be necessary. 

Finally, we must note that in education, the difficulty of 
developing "product standards** is such that the very concept may be 
meaningless. The educational literature contains examples of some of 
the more rigorous Development organizations striving to develop 
^'product specifications" prior to designing and testing prototypes. 
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However, due to the "soft" nature o£ the product in education, the 
weakness of the overall knowledge base of the field, and the general 
resistance to developing and using tight product specifications or 
standards--either in the design phase or in the course of formative 
or sunmiative evaluations--- there seems little point at this stage in 
the maturity of the educational R/D&I system to invest in development 
of product ''standards" as the solution to the issue of quality control. 
More productive may be an investment in designing Agency quality control 
monitoring processes that have within them the built-in potential for 
documentation and .analysis of Development processes and the cmnulative 
development • a stronger Development knowledge/technology base. 

3 . • r.itutional/Personnel Base for Educational Development 

The . LUtional/personnel base' for educational Development is 

weak. There aVL* some strong Development organizations, but only 

relatively few, especially in comparison to other fields (e.g.. Health). 

There a y-^rr^onnel doing Development, but relatively few personnel 

traineii in the Development process. 

The weak institutional/personnel base raises a. least three major 

issues for NIE. 

A, Contractor Selection 

On the one hand, NIE must consider whether a potential contractor 
has the kind of professional capabilities and skills to handle Develop- 
ment throughout its various phases. 

Thus, selecting a competent contractor is a key role for NIE. How- 
ever, the need for building an educational development system is so 
strong that NIE must be equally concerned about ensuring an adequate 
Development institutional and personnel base from which competent 
contractors can be selected. In many cases, this will mean that NIE 
will have to take the initiative in seeking out and developing con- 
tractors (as contrasted to simply responding to field initiation or 
selecting from among those organizations willing to invest resources 
in responding to competitive procurements). 
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B. - Quality Control of Process Instead of Product 

Given the weakness of this institutional/personnel base, and 
the difficulties inherent in either devising or using meaningful 
product standards for quality control of the Development function in 
education, NIE will do better to attempt to exert quality control 
over the Development process rather than attempt what are likely to 
be fruitless efforts to control directly the quality of Development 
products . NIE can exert some control over the Development process by 
determining how the contractor is going to approach a Development 
project, who is going to be involved, etc. 

C. The Locus of Quality Control 

Quality control may be provided in either of two ways. 

In a mature system, there will likely be a large degree of effec- 
tive field-initiated quality control. 

In an immature system (such as education), however, effective 
field-initiated control will tend to be missing. Tlius, quality control 
mechanisms will need to be set up by the Agency. (An exception can and 
should be made in those few instances where the Development institutions 
are able to and do exercise a strong degree of self-control over 
quality. ) 

In the educational context, quality control could be provided 
directly by NIE; or NIE could procure the services of other organizations 
for quality control; or, as the field matures, NIE could gradually 
facilitate the development of field self-control '.ver quality. 

We must further note that the locus of quality control will tend 
to shift through the various phases of the Development process . In the 
initial phases, NIE must play an important role -- e.g., with respect 
' to Need Identification, how Development will be done, who will do it, .. 

etc. As the Development process moves towards finalization of a 
developed product. User involvement will tend to be more important than 
NIE involvement. Thus, NIE should design and provide the mechanisms 
that will gradually shift the locus of quality control from NIE to Users. 
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D. Development in the Quasi-Public and Private Sectors 

Finally, we may note some differences between Development in 
the quasi-public and private sectors. 

IVhen Development is done in the commercial (private) sector, there 
is in the long run a built-in quality control mechanism: if the product 
is not good, it is not likely to sell for long. This is especially 
true in fields where there are sophisticated Users. Further, in the 
commercial sector, the Development/Production/Dissemination phases are 
integrated. The danger here is not so much that of selling a low 
quality product as it is selling the User products he does not need. 

In contrast, as we have been noting, when Development is funded 
by the government in the quasi-public sector, quality control cannot 
be assumed, nor is the Developer automatically linked into the total 
Development/Production/Dissemination process . 

4. Development Processes in Education 

Two aspects of Developnient as a process merit particular attention 
because of their potential implications for NIE behavior with respect 
to the field. 

A. Need Identification: Program Definition/Project Selection 

1 . So Many Needs 

It must be understood that especially in education, there are 
far mdre needs than can be met at any one time. Tliis fact necessitates 
having some form of project or program selection process. Further, 
given the limited resources of NIE, the major criterion must be cost/ 
effectiveness, with effectiveness being seen in system terms of multi- 
purposes and of portfolio effects. 

2 . Multi-Purposes/System Impact 

Any particular procurement should be examined not only in terms 
of its stated purpose, but also in terms of additional ways the project 
might (or could be used to) impact various parts of the educational 
system. For example, the project could have the additional impacts of: 
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a. building the capability of the Developer; 

b. creating linkages; 

c. helping the User during Implementation and Utilization; 

d. impacting the educational Development environment by 
helping people see educational Development in a better 
light. 

Thus, a premium should be placed on those projects which not only 
have important manifest purposes but which also can provide multiple 
additional impacts upon the total educational system. 

5. Portfolio Impacts 

Similarly the cost/effectiveness of a given project should be 
evaluated in terms of its overall impact upon and within the total 
portfolios of projects and programs at NIE and at the various Development 
institutions . 

For example: 

a. How many Development products (and what kinds) can be 
disseminated without overloading the User? Over what 
period of time? 

b. Would it be better for NIE to concentrate its resources 
in a single area in order to obtain synergistic and 
multiplier effects, in contrast to scattering its 
resources among many "good" but unrelated projects? 
Such a consideration would also suggest the need to 
examine how other (non-NIE) sources of funding were being 
used, and how funding might be coordinated across funding 
agencies to obtain synergistic and multiplier effects. 

c. To what extent would a given project tend to tie up a 
significant portion of an institution's resources? 
For how long? 

d. IVhat would a particular project do to the balance between 
high- and low-risk projects? Short-term -and long-term 
projects? What kind of balance does NIE want to have? 
What kind should a given institution have? 
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Thus, considerable eraphasis should be placed on those projects 
which fit best into the total portfolio of projects. 

Other illustrations could easily be given of multi-purpose and 
portfolio impacts. The point is simply that project selection should 
be considered not just from the viewpoint of a single project in 
isolation but also from the viewpoint of total system impact. 

4. NIE and Field Roles in Project Selection 

Viewed from this above perspective, project selection is not merely 
the evaluation of proposals. Even more significantly, project selection 
is a key element of system orchestration, in Development but also (of 
course) in other R/D5I system functions. 

Further, from this perspective, it becomes possible for NIE to 
think of potential divisions of roles between NIE and the field. 

Specifically, the field, should be involved primarily at the 
point of proposal evaluation, while NIE must take responsibility for 
project and system orchestration. 

The field should be in the best position to evaluate whether an 
innovation is "good" or "bad," at least in terms of such key criteria as: 
relevance to User need; User capability to implement and utilize; state 
of the art; User climate for acceptance; etc. Parenthetically we may 
note that to the extent NIE lacks education professionals on its staff, 
NIE will be dependent upon field evaluation. This dependence is not 
an issue here. By allowing the field to provide evaluation from its 
perspective and capabilities, NIE is freed from some unnecessary 
additional burdens and can focus its resources on: a) the evaluation 
of proposals in terms of orchestration and total system impact, and 
b) doing the actual orchestration. At the same time, we do not want 
to downplay the importance of having education professionals on the 
Institute's staff. They would seem to be needed to help give an edu- 
cation perspective to NIE's orchestration rolv^. 

The field cannot really be of much help in making orchestration 
and total system impact evaluations simply because they are not likely 
to be in touch with (or even know about) all the elements of the NIE 
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portfolio; to know which institutions need building; what funding is 
available from what sources for what projects; etc. In other words, 
the field has neither the system information nor the system perspective 
to make a total system evaluation or to take responsibility for or- 
chestration. 

Finally, we may note that the field's lack o£ a total system 
perspective may well be a strong point in favor of NIE permitting the 
field to play a major role in evaluating individual proposals Since 
the judgments of NIE personnel are likely to be affected by their con- 
cerns about system building, etc., evaluations of individual proposals 
by the .field provide the Agency with a rather different set of evalu- 
ative judgments to input into the project selection process --judgments 
based on the merits of individual proposals alone, unhampered by the 
other kinds of concerns that would arise inevitably from application of 
a total system perspective. 

S . Potential Equivalency of Alternatives 

In Development (as elsewhere to some degree), it is not obvious 
and clear that one product will be of more benefit or is more necessary 
in the long run than another product. InnDvation rarely comes from a 
once-and-for-all "great breakthrough." Rather, significant innovation 
generally results from a series of small improvements, and such im- 
provements may result from a myriad of alternative approaches, methods, 
projects, etc. 

Thus, NIE will always be in the position of making cost/ef fective- 
nei>3 orchestration judgments , not only between proposals, but between 
the need to orchestrate the field, on the one hand, and the value of 
a given innovation, on the other. And NIE will not have the luxury of 
certainty about its judgments. This does not mean that NIE should avoid 
the responsibility of making cost/effectiveness judgments by relying on 
others (e.g., a panel of "experts" from the field). As we have already 
noted, the field cannot orchestrate. Rather, the Agency staff, after 
obtaining the best information it can, must simply accept the responsi- 
bility to make (and live with) orchestration judgments. 



6. The Extent of NIE Influence in the Field 



In terms of funding, NIE is only one of many sources which fund 
education or education-related Development. NIE is not even the 
largest funding source. Further, NIE does not have the power or status 
of such other funding agencies as the National Institutes of Health. 

Nonetheless, since NIE is the lead agency for Research and Develop 
ment in education, its orchestration role is vital. Futher, even 
though NIE may not have a large direct influence on the field, NIE 
can have a significant indirect impact on education through its orches- 
tration role, by tailoring its efforts carefully in terms of such 
criteria as: knowing what is and is not being done; knowing where the 
strengths and weaknesses of the system lie; knowing where it can and 
cannot help. Indeed the combination of NIE's relative smallness and 
its having the lead responsibility make . its orchestration role even more 
important . 

7. Appropriate Evaluators 

Selection of personnel to evaluate proposals, projects and product 
is a key element for an overall NIE Development project selection strate 
Determining at what point (during the overall Development process) each 
evaluator has an appropriate role is also important: 

1. Users have a valid evaluation role at three main points: 

a. early in the Development process, to identify their 
needs; 

b. at the point of proposal evaluation, to help NIE 
evaluate the specific merits of individual proposals; 

c. later in the Development process, as products begin 

to enter the developed stage; and then as products are 
implemented and utilized, so as to provide corrective 
feedback to Developers and/or Producers. 

2. Developers should have a role early in the process to help 
identify what projects should be worked on, especially to 
provide some input into decisions about what is possible 
given the Development state of the art. 



3. Researchers will play a very minor role in the evaluation 
process. Where the state o£ the art leads to a very large 
degree of uncertainty, Researchers might be able to provide 
some insight. 

4. NIE has two basic evaluation roles: 

a. NIE must evaluate proposals from the total system 
impact perspective (multiple purposes and portfolio 
impact) . 

b. NIE must also select the evaluators from the field, 

both in terms of individual capabilities and perspectives 
and of providing balance in the field evaluation. For 
example, if there is to be an evaluation panel, it is 
NIE's responsibility to determine how many and what 
Users, Developers, Researchers; at what different points 
in time; organized in what ways; etc. In other words, 
the use of field evaluators should be a carefully thought- 
through and planned process, one which can vary according 
to situation. 

8. Sources of Need Identification: Practice v. Technological 
Opportunity 

Thus far we have been emphasizing practice-based needs — i.e., 
the needs identified by Users, which are based on their everyday ex- 
perience. It is also important to consider needs which are technology- 
based--i.e., when it becomes technologically possible to develop a 
product, consideration must be given to developing (and in some cases 
even promoting strongly) that product. 

The issue now becomes: What is a proper portfolio balance between 
needs as currently understood by the field and responsiveness to what 
has recently become technologically possible.' On the one hand the 
"bread and butter** type of Development is generally based upon User 
identified needs. On the other hand, it is likely that the accumulation 
of Development and Research knowledge will at times make possible the 
development of b very significant product. Thus, NIE will want to 
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orchestrate a portfolio that permits strong technological oppor- 
tunity types of Development work to be supported. 

The implication for the NIE-orchestrated selection process is 
that Users will predominate in the evaluation of User-based needs, 
and Developers (with some help from Researchers) will predominate 
in the evaluation of technological -opportunity-based Need Identi- 
fication. 

9. Interrelationships and Orchestration: Some Implications 
To summarize some points we have been making: 

1. Consideration must be given to potential synergistic 
effects across projects that may result from develop- 
ing a portfolio of interrelated projects—as contrasted 
to a lack of synergistic effects from an unconnected 
set of unrelated projects. 

2. Consideration must be given to work that is supported 
elsewhere (e.g., by the Office of Education, the 
National Science Foundation, other non-government 
foundations) . Here the relevant issues would be (for 
example) : 

a. How can the various funding sources pool resources 
and orchestrate efforts to develop synergistic 
effects across projects? 

b. How adequate are the non-NIE sources of support 
in a particular area? Are they such as to allow 
NIE to give its attention and support to other 
areas? 

B. Two Basi- Operating Modes in Educational Development 

Basically, there are two rather different operating modes 
through which the Development function is carried out in 
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education — Development work as it is generally carried out 
in specialized Developiaent organization, and Development 
work as it tends to take place in practice-based (or 
practice^related) settings . 

1. Specialized Development Organizations : 

The Development function, as it is carried out in pur- 
suance of government contracts — primarily in federally- 
funded regional laboratories and R§D centers and in 
some of the non-profit and for-profit Research and 
R5D coiporations — tends to adhere, at least in form, 
to the engineering model of Development used in indus- 
try. 

Development in these settings is an institutionalized 
specialty carried out by specialized personnel in 
specialized Development organizations or organizational 
units. In the best of these organizations. Development 
activities tend to be systematic and sequential, moving 
in a smooth progression from the prototype design that 
is the end product of the Research phase of R/D§I, to 
product or program Development in accordance with detailed 
specifications, to evaluation of small field tests, to 
revisions, to larger field tests, to more revisions, to an 
additional field test, etc. until the product performs in 
accord with prespecified performance objectives. Products 
go through successive generations of revisions, each a 
closer and closer approximation to the performance specifi- 
cations. Revisions are based on empirical field test data 
that are gathered systematically and analyzed rigorously, 
and the evaluation data are expected to provide the poten- 
tial User with information about the outcomes or effects to 
be expected from use of the product under specified Im- 
plementation conditions. 
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Clearly, there is a considerable amount of Develop- 
ment work carried on in these organizations that does 
not adhere faithfully to this relatively rigorous model 
of the Development function. But equally clearly, this 
does appear to be the model they are trying to use in 
planning and conducting their work. 

Development projects implemented in accord with this 
model tend to be large-scale and exp'ensive, involve large 
personnel pools and heterogeneous skill mixes, and require 
extensive cooperation between the organization developing 
the products and the school systems agreeing to serve as 
field test sites. The products themselves are often com- 
plex, consisting of many and varied mod-ales or components, 
and often several forms of media as well as printed mater- 
ials. The management of these complex projects is often 
highly formalized, using flow charts and sophisticated 
management tools. 

There are some variations in pattern depending on the 
nature of the products being developed--e.g. , products vs. 
change processes. But the issues of concern to managers 
tend to be consistent--e.g. , How much Research is needed 
prior to the Development work; how much Research can 
proceed parallel to the Development work? At what point is 
the product sufficiently developed to permit initial field 
testing? At what point has the product been tested suffi- 
ciently to permit Dissemination? What Dissemination, Market- 
ina, and Implementation factors need to be considered 
thfe.jughout the design and development phase? At what point 
does the responsibility of the Developer end--Development? 
Dissemination? Installation? Utilization and Maintenance? 

These issues are to some extent common to the Development 
function in all sectors, but they take on particular signi- 
ficance in education. The weakness of the knowledge and 
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technology base of the field makes it more difficult to 
translate performance specifications into effective 
products. Outcomes are much less predictable given: 
a) the nature of the interaction between User and product 
and b) the limited technical capability of Users to imple- 
ment complex innovations without substantial Implementation 
supports and/or assistance. Consequently, Development work 
in education requires a far greater investment of time and 
money in the Research and Evaluation components of the 
R§D process, making Development costs high relative to 
practical a problem of particular importance 

considering the negative political climate in which edu- 
cational R§D appropriations tend to be made. 
Practice- Based/Practice-Related Development: 

A rather different mode of Developraent is inherent in 
traditional approaches to the design and development of 
instructional strategies and materials--as these activities 
have been carried out by classroom teachers, by curriculum 
specialists in SEAs, L£As, and universities, by publishers, 
and by the university scholars who have on occasion par- 
ticipated in efforts to improve K-12 level curricula and 
instructional materials in their areas of specialization. 
We note in particular the significant amount of program 
design and development that has been carried out within 
LEAs in response to the availability of categorical 
funding from federal (and some State and other) sources — 
e.g., ESEA Title I and Title III funding from the Office of 
Education . 

The Development approach used in these settings tends to 
be intuitive rather than data-based or grounded in in- 
structional theory. The focus of attention is generally on 
the content to be conveyed rather than on conceptions of 
how teachers go about providing instruction. Field-testing 
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is non-existent or minimal. Whatever evaluation is 
carried out tends to focus on the face validity (e.g., 
expert judgments, teachers* subjective perceptions and 
reactions) rather than measured effectiveness in achieving 
prespecified impacts. There may not be a systematic 
testing program or rigorous analysis of empirical data. 
In comparison to the Development mode uhat characterizes 
specialized Development organizations, there is less likely 
to be a systematic cyclical test and revision sequence. 
Even where some evaluation and revision does take place 
there is less likely to be extensive recycling, and it is 
highly unlikely that evaluations would be made in terms of 
a product's effectiveness in achieving prespiecified measur- 
able objectives. 

There are other significant contrasts between the two 
Development modes. In comparison to the pattem in special- 
ized Development organizations. Development costs in 
practice-based Cor practice-related) settings tend to be 
relatively low. The perso'nnel involved are relatively 
few Ce.g., one teacher, a few scholars or curriculum 
specialists, etc.) and whatever skill mixes are present in 
a Development team tend to be relatively homogeneous. 
Management is generally informal and highly flexible. 

Within the practice-based/practice-related mode, two 
important variants can be identified, depending on whether 
or not the product or program of interest has been developed 
with the intent of wide-scale dissemination in mind. Where 
textbooks or materials packages are being developed for 
large-scale, nationwide dissemination, an effort is usually 
made to include Implementation supports in the form of 
teachers' guides, tests, etc. Where materials are developed 
locally within the User system for use by a single teacher 
or group of teachers in a single school or District, far 
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less of the Implementation process is committed to print 
or media presentation, the state of "development" of the 
materials or strategies -for use outside this small group 
remains inadequate, and either the locally developed in- 
novations are not disseminated at all or they are dissemin- 
ated but have minimal success elsewhere because Develops at 
work was not carried far enough to permit the materials to 
be implemented easily and effectively by others. 

The exemplary practices or programs that have become the 
focus of so much attention in recent federal and State 
Dissemination programs fall within this practice -based mode. 
Where exemplary practices or programs axe neither designed 
in a form that permits them to be generalized to other 
schools or Districts, nor packaged in a form that permits 
dissemination, these practices/programs may provide poten- 
tial inputs for the Development function as this is carried 
out in the specialized Development organizations. We shall 
return to this point shortly. 

Strengths and Weaknesses of the Two Basic Modes 

Given that Development work in education tends to be carried 
out in one or the other of these two basic modes, our first 
concern must be to identify the strengths and weaknesses of 
each mode. We will then be in a position to identify several 
issues with significant implications for NIE behavior, 
a. Specialized Development Organizations ; 

The specialized Development organizations in the edu- 
cational R/D§I system have (at least potentially) several 
basic strengths. They are generally large in scale and 
tend to be staffed by sizeable numbers of skilled personnel 
(quite possibly) at the "minimum critical mass" level. 
These personnel are more likely than the practice-based 
and practice-related Developers to be knowledgeable about 
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the Development state o£ the art. These organizations 
tend to have the capabilities needed to undertake large- 
scale projects and are generally able to carry out all 
steps of the Development process. 

These organizations also have several potential disad- 
vantages. Because they are external to the User system, 
they may tend to be isolated and insulated from User needs- 
working, as it were, in an "ivory tower", setting. Dis- 
semination and Implementation may be problematical because 
of inadequate linkages to the User population and/or to 
intermediary organizations, because their products may 
not be sufficiently relevant to User needs, and because 
the User may react to these externally developed products 
in terms of the all-too-familiar "not invented here" 
syndrome. Given the high costs of rigorous Development 
work in these specialized organizations, the relatively 
limited utilization of their outputs to date in User set- 
tings, and increasing recognition of the extensive amount 
of local innovation that exists in some parts of the User 
system — questions continue to be raised about the relative 
efficacy of this Development mode. If this mode is to 
remain viable it would seem essential for NIE to relate to 
these organizations in a manner that strengthens their func 
tioning. 

b. Practice-Based/Practice-Related Development : 

The obvious advantage of the practice-based/practic^J- 
related mode is its general closeness to real User needs 
and problems, especially in those cases where the Developer 
is the User or is at least part of the User system. 

However, several disadvantages of this mode are* also 
obvious. For example: 

1. This tends to be an inefficient and at times in- 
effective mode. Since the personnel carrying out 
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this kind o£ Development work tend to be part-time 
Developers at best (and often people who may carry 
out Developmeat work only once or twice or veiy few 
times) — personnel who by training, background, and 
commitment tend to be identified with other kinds of 
activities — there is limited likelihood of these 
Developers being very aware of the Development state 
of the art. Thus, there will tend to be much '*re- 
invention of the wheel"--duplication of effort, etc. 
Further, most of the Developers who carry out this 
type of Development work lack both the skills and 
time needed for sophisticated and complex develop- 
mental activity. (One important exception here are 
the Production personnel who generally work for 
textbook publishers. But even in the publishing 
example, the university scholars or education 
practitioners who may do the bulk of the conceptual 
and substantive, as opposed to editorial and Pro- 
duction work, are likely to fit the above description.) 

2, Even when the Development output appears to be high 
in quality, there is generally little accompanying 
empirical evaluation data to serve as the basis ^for 
making quality control judgments or Adoption/ 
Adaptation/Implementation decisions . 

3. When Development work is carried out within the User 
system--whether in classrooms, curriculum committees, 
teacher centers, LEA or SEA offices--the output (a 
curriculum, a set of materials, etc.) is being 
developed for a particular local component of the 
User system and is generally not automatically input 
into some broader Dissemination system able to make 
it accessible to a larger population of potential 
Users • Thus, there are rather substantial problems 
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encountered in identifying and disseminating high 
quality developmental work carried out in these 
settings. 

4. There is one other significant problem related to 
the fact that this kind of output is generally 
being developed for only a local component of the 
User system--by a User-Developer for personal use, 
or by a Developer who can interact personally with 
a relatively small number of Users (e.g., an LEA 
curriculum coordinator, or a teacher working with 
other teachers in her school or District) . More 
often than not, the kind of Implementation informa- 
tion that a User-Developer is likely to carry around 
in his/her head is not elaborated in print or through 
media presentations. For this is the kind of in- 
formation that the User-Developer generally feels 
need *; - ^e committed to paper for personal use, and 
can be readily communicated orally in interpersonal 
exchanges with other Users in the school or District 
with whom he or she may share the Development output. 
If this kind of output is to be disseminated more 
broadly it must be packaged properly — a problem usually 
beyond the time and skills (and probably the interest) 
of the individual Developer-User. 

c. Issues 

In relating to the specialized Development organizations 
in the educational R/D&I system, the key issues for NIE 
to consider would seem to be: 

1. the extent to which NIE should specify the substance 
of, and/or provide ^direction To, their Development 
activities; 

2. the extent to which emphasis should be given to 
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system building and/or rebuilding; and 
3. how to overcome the Dissemination weaknesses in- 
herent in this operating mode based in settings 
external to the User system, 
"^r relation to practice-based/practice-related Develop- 
ment work, the key issues for NIE would seem to be: 

1. to what extent these outputs should be validated, 
and what mechanisms might be most appropriate for 
whatever degree of validation is seen as needed; 

2. what mechanisms might be most appropriate for 
identifying high quality materials, programs, 
practices, etc. developed within the User system; 
and 

3. what strategies might be most appropriate for 
supplementing the efforts of the Developer-User at 
various steps in the Development process that are 
not likely to be carried out adequately (or at all) 
without the intervention of some State or federal 
agency or some intermediary organization (e.g., 
the packaging. Production^ and Dissemination of 
exemplary programs or practices created by User- 
Developers) . 

It is woth noting that Dissemination is an issue for 
both of these two basic modes of educational Development. 

Given that these are two very significant modes of 
educational Development, the overarching appears to be: 
on the one hand, how to capitalize on their strengths 
while on the other hand, how to build into the system 
countervailing forces to overcome their weaknesses. 

4. The NIE Role 

It can be seen from the above delineation of issues that 
NIE must accept responsibility for orchestrating the 
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- linkages — simply because the need is there and there is 
no other agency with the authority (or perhaps too the 
inclination) to do thi'' orchestration. 

In relation to t)\ large-scale Development organiza- 
tions, there is a need to develop the linkage between 
these organizations and Usors. NIE should be directive 
in developing and controlling the linkage process . 
That is, NIE should insure that the process leads the 
Developer both to develop products that are needed and 
to disseminate these products. This is related to our 
earlier point about the needed User role in Development 
program definition and project selection. 

In relation to the practice-based mode of Development, 
the needed NIE role would seem to be to develop and/or 
facilitate mechanisms which would permit the identifi- 
cation of exemplary practices and other outputs which 
could and should be generalized, packaged and dissemina- 
ted (and probably validated as well) . The question now 
arises: who should begin the process by identifying 
exemplary practices, etc. for input into more formal- 
ized and rigorous developmental activity, e.g., pack- 
aging and validating? On the one hand. Users could 
identify such practices and bring them to the attention 
of some other (probably intermediary) organization'i . 
On the other hand, intermediar)' organizations could 
bear the brunt of these "search and discover" opera- 
tions . 

Once such exemplary practices h£LV<> been identified, 
they must then be recast in forms that are generaliz- 
able to other settings, and then, they must be packaged, 
produced, and disseminated. 

Where organizations .with^ the needed capabilities 
already exist in the field, NIE's role would seem to be 
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to facilitate their work (and where possible their 
growth) by providing resources and/or making it more 
feasible in other ways for them to function better and 
to grow. 

However, if the needed organizations do not exist 
(or are too few), NIE's options would seem to be to 
either "buy" or "rent" the needed capabilities. The 
"buying" strategy would involve a direct attempt to 
create insitutions that would become a permanent part 
of the educational Development system. A "renting" 
strategy would entail the temporary purchase (there- 
fore, the term "renting") of services from organizations 
that already possess the needed skills but are external 
to the educational R/D§I system. In this case, a 
"gap" in the system is "filled," but only temporarily, 
on a project-by-project basis. The advantages of the 
"renting" option are that the packaging, production, 
etc. can be done quickly, NIE can be rather directive 
about what is to be done, and the funding need be for a 
specified (short-term) period only. The disadvantage 
is that this option does not increase the overall long- 
term capabilities of the educational Development system. 
We shall return to this point later in our analysis. 

Thus far we have discussed possible NIE roles and 
options vis a vis Development organizations and practice- 
based Developers as separate, unrelated courses of 
action. However, if NIE were to pursue a "buy" strategy 
for packaging exemplary programs, practice-based 
Developers and Development organizations could be linked 
in such a way as to improve the overall, long- terra 
capabilities of the educational Development system. 
Here, the role of NIE could be to create special link- 
ages between Users and Development organizations such 
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that the flow of exemplary practices into the 
Development organizations for packaging and dissemina- 
tion would become a normal and permanent part of the 
Development system. Initially, we would anticipate 
that the bulk of the exemplary practices that would 
flow into the Development system for packaging and 
dissemination would be identified by intermediary or- 
ganizations specifij?aH)i' assigned the task of identi- 
fying exemplary programs, practices, approaches, etc 
operational in the User system. Over ti:ne, howeve., 
we would anticipate an ^ncrease in User- ^ni tl^^ed 
communication of exeiTiplary practices to the Develop- 
ment organizations. This linkage and communication 
flow might be strengthened over time as the outputs 
of these Development organizations become increasingly 
User- relevant and high-quality. Under such conditions, 
we might hope for a -greater User awareness and appreci- 
ation of the role of these Development organizations. 
Thus the real issue is: 

1. whether NIE wants to insure that many exemplary 
practices are identified, packaged, and dis- 
seminated quickly (and therefore might choose 
the "rent" option) ; or 

2. whether instead NIE prefers to build slowly and 
facilitate development of capabilities and 
linkages that will become permanent and expand 
the capacity of the overall system (and there- 
fore might choose instead to function in a "buy" 
mode with built-in system building potential). 
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III. IMPLICATIONS FOR NIE BEHAVIOR 

1 ■ Procurement Behavior 

The key theme throughout this analysis of educational Development 
has been that NIE is not simply a "channel" for funds. Rather, NIE has 
a major orchestrating role because the context calls for this type of 
role, the need is so great, and there is no other institution available 
to carry out this role. Further, orchestration implies not simply a 
balancing of types of products or programs, but also — and more important- 
ly — a balancing of purposes. As noted in our introductory chapter, 
when NIE makes a procurement, it may procure: 

1. a product or a program; 

2. system building; and/or 

3. some change in the environment of the educational 
system. 

Whether it intends to do so or not, whenever NIE makes a procurement, 
that procurement may affect one, two, or all three of the above purposes. 
Thus, for example, a procurement made with the manifest purpose of 
permitting the rapid design of a given product may achieve that parti- 
cular purpose and have little if any discernible systemic impact. 0£, 
the procurement may achieve the manifest product-design purpose while 
at the same time also strengthening (or weakening) the long-term 
capabilities of a particular part of the educational Development system. 

A. The "What" of Development Procurement Orchestration 

Throughout this analysis, we have noted a number of specific 
items which NIE can and should orchestrate through its Development 
funding policy. For example: 

1. obviously, a specific, relevant, and usable Development 
product; 

2. system linkages (e.g., between Users and Developers, be- 
tween Developers and the Development state of the art); 

3. balance among and between multi-purposes and portfolio 
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effects ; 

4. coordination and synergistic effects between NIE and 
other funding agencies and/or combinations or projects; 

5. system building and rebuilding. 

B. The "How" of Development Procurement Orchestration 

We have noted a number of options or alternative strategies 
concerning the methods of orchestration. 

1. NIE can operate in a consensus building mode, i.e., pro- 
viding the mechanisms whereby field personnel can come 
together to reach agreement as to key issues, directions, 
etc. Basically this is a process of orchestration through 
the use of field personnel for inputs to key planning 
decisions. However, we note that NIE must orchestrate 
the selection both of which field personnel and which 
mechanisms are to be utilized, and in relation to what 
issues or concerns. 

2. NIE can choose to orchestrate through the prope:; selection 
and p lacement of advisors in relation to specific aspects 
of the Development function. That is, NIE can orchestrate 
procurements through decisions about who will be involved 
in Need Identification, in various stages of quality 
control, in making decisions about field testing and pack- 
aging, etc. 

3. Orchestration may be carried on through staging the procure- 
ment process— either by using different contractors for 
different stages of the Development process (e.g., differ- 
ent contractors for Need Identification, designing, field 
testing, etc.); and/or funding only one stage at a time, 
reviewing after each stage, and then continuing, modifying 
or terminating the process. 

4. NIE has choices to make among the strategies of facili- 
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tating, buying or renting. We will discuss this more 
fully below. 

Regardless of which combination of the above options 
NIE chooses, there are two aspects of orchestration method- 
ology which our analysis has emphasized. These are: 

5. Orchestration requires a monitoring process- Thus, moni- 
toring becomes a critical NIE activity that has to be 
continuous and* ongoing. 

6. Given the nature of the educational R/DSI system as a 
whole and of the educational Development system in par- 
ticular, orchestration of process is likely to be more 
fruitful (and even feasible) than directive control of 
products . 

C. Facilitating. Buying or Renting 

NIE can choose among facilitating, "buying," or "renting" 
strategies, both within projects and across its total portfolio 
of Development projects. We discussed these options earlier in 
connection with the packaging, production, and dissemination -of 
exemplary practices developed within the User system. The same 
points could apply equally well to other kinds of Development 
activities. 

As noted earlier, though, it should be underscored that the 
choice among these options must be based in part on the kinds of 
capabilities that do or do not exist at a given point in time 
within the educational R/D5I system. A facilitating strategy is 
possible for NIE only where the needed capabilities exist to 
some degree or other within the educational R/D5I system. If the 
needed institutions do not exist (or are too few), the facilitative 
strategy is irrelevant. Thus, NIE would then have to create the 
needed parts of the system, either on a permanent basis (buying) 
or on a temporary basis (renting) . 

Each of these strategies call for a somewhat different manage- 



ment mode by NIE. Where a facilitating strategy is possible, the 
collaborative mode of management is most appropriate. The buying 
and renting strategies, in contrast, call for and permit a more 
directive mode of management by NIE, with renting requiring/ 
permitting the most directive approach. 
Three points should be made here: 

1. Both buying and facilitating are system-building modes. 
Renting is not, though some system-building capabili- 
ties can be built into a renting strategy. 

2. In general NIE should seek to build the educational 
Development system (though there are exceptions, as 

we shall note), and where possible it is best to build 
through the facilitation of existing institutions. 

3. NIE has been receiving some criticism from the field 
for behaving in a manner that some in the field view 
as overly directive. While we, in this analysis have 
been emphasizing a process rather than a directive 
management mode for NIE, we must also note that at times 
NIE may have no choice but to be directive. A facili- 
tative mode, by definition, is relevant only when con- 
ditions in the field permit. When there is a lack of 
adequate capability in the field, it is this weakness 
which requires the taking of a directive approach. 

Some additional comments would seem to be in order 
about the renting strategy. As already noted, renting 
is not a system- building strategy. That does not mean 
renting is by definition a "bad»» or ''lesser of evils" 
strategy. For one thing, the nature of Development is 
such that there may be a large number of specialized 
' tasks. Where a task is highly specialized, and/or seldom 
used, renting may indeed be the "best" strategy. There 
is nothing inherent in the Development process that re- 
quires all tasks to be performed by organizations within 



the Development system of a particular sector, 

A second point is that renting can be very helpful 
as an interim strategy — either because it is simply 
necessary, or because at a given time NIE is not yet 
certain whether there is or is not a long-term need for 
building a specific internal system capability. 

Nonetheless, where renting is used as an interim 
strategy, NIE must recognize that renting can retard 
system development. If NIE wants to develop a particular 
internal system capability in the long run, there is the 
danger of never starting the process of building simply 
because the products being rented look so much better 
than what initially could be obtained within the system. 

Thus, where renting is an interim strategy, there 
should be some mixing of renting and buying strategies. 
For example, the rental contract might mandate a certain 
amount of training for personnel within the educational 
Development system. 

Finally, even when renting is a valid strategy, there 
are factors which will tend to constrain.NIE's willing- 
ness to use the strategy— e,g, , a high price, a large 
amount of sub-contracting, etc, 

D, Excellence in Development 

We should note here that while "Excellence'' may be considered 
a desirable procurement objective, "Excellence" has a meaning in 
Development rather different from its meaning in Research — a 
difference that significantly affects funding policies and decisions. 

In Research, "Excellence" is measured against the best the 
field can do working at the outer limits of the state of the art. 
Thus, in Research, "Excellence" implies creativity, the utmost in 
understanding and knowledge, even "breaking through" or expanding 
the limits of the state of the art. In Research, "Excellence" is 
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not measured against practical considerations such as cost,, time, 
operability, availability, usability, etc. 

In contast. Development "Excellence" i£ measured by these 
practical operational criteria and is not measured by the "outer 
limits" of the state of the art. 

For example, in Development there may be a choice between two 
potential contractors. The first can develop a product which is 
"good enough to do the job," and can do so at a reasonable cost 
and within a reasonable period of time. Further, to fund this 
organization would result in strengthening the overall Development 
system. By contrast, the second organization can develop a product 
which is potentially superior and certainly more creative, but 
can do so only at a much higher risk and cost and it may take much 
longer. Further, funding the second organization would not result 
in any overall or long-term strengthening of the Development 
system. In Research, the wise policy would be to fund the latter 
organization. In Development, however, it may be a wiser policy 
to fund the first of these organizations. 

In Development, usability is the key criterion of Excellence- 
Usability, costs, time-frame and system-building would seem to 
be the criteria most appropriate for project selection, 

E, The Effects of Funding Policy on Development Institutions 

NIE should keep in mind that its funding policy toward Develop- 
ment institutions tends to mold the character of these institutions. 
It is the nature of institutions (particularly the more mission- 
oriented) to tend to become what their funding sources (on whom 
their exis^tence depends) demand of them in terms of kinds of 
products, kinds of skills, kinds of risks, etc. It is, in simple 
terms, a case of "He who pays the piper calls the tune," This 
dynamic is especially true of Development organizations, for 
they, in sharp contrast to Research organizations, tend to be 
able to acquire rapidly the skills required by the types of 
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projects funded. For example, in comparison to Research organiza- 
tions. Development organizations tend to have a clearer division 
of labor, tend to acquire personnel specializing in less creative 
areas of work, which require less knowledge, etc. 

A related concern is whether (and to what extent) NIE should 
fund Development organizations on a programmatic as contrasted to 
an institutional support basis. NIE has been grappling with this 
issue since the beginnin^g of its existence. The tide now appears 
to be moving back noticeably toward acknowledging the need for 
some degree of institutional support for organizations with 
which it has (or chooses in the future to have) a "special rela- 
tionship"--or at least some kind of balance between institutional 
and programmatic support. 

We note this concern because (in contrast to Research or- 
ganizations) Development programs which are government-funded 
tend to allow very little room for interpretation or maneuvering. 
When a Research program is funded, there is a sufficient level 
of uncertainty involved to permit the Research organization to 
choose from among a variety of alternatives and still remain 
within the program's guidelines. By contrast, a specified 
Development product tends to be defined more narrowly and the 
Developer's options are considerably more limited. 

Thus, for the overall health of a Development organization, 
some institutional funding would seem to be essential, whether 
this be 10% or 20% or some other figure. Otherwise, the creative 
aspect of that organization will be stifled and its character 
molded to a very large extent by the funding agency. 

F. Competitive Bidding vs. Sole Source 

,We must now call attention to an issue which is, to an im- 
portant degree, political. Government policy favors competitive 
bidding for government funding, and for very valid reasons. 
However, the very concept of orchestration (especially in the 
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context o£ an immature system such as educational R/D5I in general 
or educational Development in particular) in5)lies that there will 
be a significant (probably predominant) need for sole-source 
type of funding. 

Orchestration requires the ability to control the various 
parts and aspects and purposes of system funding. By definition, 
competitive bidding takes away from tiie funding agency a signi- 
ficant part of its ability to orchestrate. Even when there is 
competitive bidding, the ability of a potential contractor to 
"fit into" the funding agency's orchestration efforts may have 
to be one of the criteria for the contractor's being permitted 
to submit a bid and/or for the selection of one bid over another. 
For example, if one of NIE's orchestration objectives is to build 
a few strong Development organizations (as compared to funding 
many weak institutions with little or no system-building effects, 
given limited NIE resources), a fully open competitive bidding 
policy for each individual product or project may well negate 
such NIE orchestration efforts and capabilities. 

G. Single vs. Multiple Institutions 

Thus far, our discussion may seem to imply the funding of a 
single institution to carry out a particular project. This need 
not be the case. There can be good reason to fund several organiza- 
tions to work on a single project. For example: 

1. No one institution may have all the capabilities needed on 
a given project. For example, one may be strong in Need 
Identification because of strong ties with Users— but have 
lesser product design skills. Rather than eliminating that 
organization from the project, an option could be to fund 
a second organization for the product design stage. An- 
other example could be a product requiring the perspective 
of different disciplines (e.g., psychology and sociology as 
well as education). Under such circumstances, NIE might 
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choose to fund a collaborative relationship between two 
or more organizations with the necessary perspectives 
and skills. 

2. Collaboration between two or more organizations could 
have synergistic effects- 

The use of several organizations could take a number of forms. 
For example: 

1. Staging . Different organizations would be funded for 
different stages of the Development process. Here the 
role of NIE would be to orchestrate and facilitate the 
transitions between stages and organizations. 

2. Collaboration . Several organizations would be funded to 
work together on the same stage or stages of the Develop- 
ment process. Here, NIE^s role would be significant in 
selecting the organizations and perhaps facilitating 
their collaboration. 

3. Supplementing . Here, one organization would have primary 
responsibility but would need some supplemental help in 
one or more aspects of the process. Here, the NIE role 
could be either to select an organization which has con- 
nections with "satellites," or to select the supplemental 
org£inizations . 

4. System building . Here, a weak but potentially valuable 
organization could be "paired" with a stronger organiza- 
tion. The "weaker" organization could thus gain experi- 
ence. Here, the primary NIE role is selection. 

5. Funding Coordination . Where other funding agencies either 
are or could be involved, the use of more than one Develop- 
ment organization might simply be a political necessity in 
order for funding to be coordinated. Further, the use of 
multiple Development organizations could be seen by the 
various funding agencies as having a potential synergistic 
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benefit • 

Regardless of the form, use of multiple, organizations would 
necessitate some form of orchestration by NIE. 

2. Non- Procurement NIE Activities 

Thus far we have tended to focus on the procurement ac- 
tivities of NIE, i.e., those actions which lead to the award 
of (and invovle Agency personnel in monitoring) grants or 
contracts for various R/D§I outputs. Indeed, if one were to 
analyze NIE's activities solely in terms of its expenditures, 
it might well seem that the Institute's role is defined 
largely in terms of procurement activities. 

However, there are various other kinds of activities 
which NIE could initiate, or facilitate, and we have mentioned 
some of these at various points in our analysis. Examples 
of these non-procurement activities include: 

1. Coordination and collaboration with other agencies 
funding Development activities ; NIE is the lead 
agency for Research and Development in the education 
sector, and is the only existing agency with a legis- 
lative mandate "to build an effective R§D system" 
for education. Thus, it is the only organization 
that can legitimately claim some "right" to function 
in a system orchestration role. 

2, Conferences and seminars : We have noted several 
kinds of communication linkages which, are weak or 
missing in the educational R/D§I system. Conferences 
and seminars provide one way to provide some communi- 
cation between groups and to develop the informal 
communication networks which tend to exist in more 
mature R/D§I systems. Such conferences might bring 
together: Users and Developers (to improve the Need 
Identification process, or to provide Developers with 
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linkages to exemplary practices in the field) ; 
regional groups of Users (with a focus on either 
some specific topic of relevance to the region or 
simply beginning the development of informal region- 
al communication linkages); Developers (again, to 
help Developers become more aware of possible project 
linkages among different Developers, and to begin 
development of informal networks); etc. 

Considerations Internal to NIE 
A. NIE Staffing and Development 

The educational issues and potential NIE roles which we have 
been discussing have implications for the kinds of staff NIE needs, 
the kinds of skills its staff must have, and the kinds of structures 
which are or are not relevant to NIE. 

While the scope of this study does not permit full delineation 
of these implications, some observations may nonetheless be made. 

1. The emphases of this analysis on: a) orchestration as a 
major role of NIE and b) process facilitation as a basic 
management mode call for organizational structures and 
management styles and skills other than those normally 
associated with a bureaucratic type of organization. 

2. Further, these same emphases, in the context of the im- 
maturity of the educational R/D§I system, call for staff 
personnel in NIE who are educational professionals and who 
continue to be actively involved in various aspects of 
the field. 

3. A somewhat decentralized rather than highly centralized 
type of organizational structure would seem to be better 
suited to gaining entry into the User system, to develop- 
ing linkages with and between various parts of the edu- 
cational system, and to facilitating system building within 



the field- 

B. NIE^s View of Its Relation to the Field 

Most of this £inalysis has focused upon an orchestration role 
for NIE, The question at this point would seem to be: Does NIE 
accept that role for itself? Or does NIE prefer to play a 
different role? 

If NIE is willing to accept the orchestration role, then the 
Institute must begin the process of developing the kinds of goals, 
priorities, policies, structures and staffing that are relevant 
to and/or required by this role. In this process, it may very 
well be that a certain amount of internal re-orientation and re- 
training would be wise and/or necessary. We shall return to these 
considerations later in our analysis. 
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I. THE GENERIC CHARACTERISTICS OF DISSEMINATION 

1 . Introduction 

In the sense of a segmented conceptual model of a total R/D{il 
process. Dissemination may be considered as a separate, distinct 
part of the process-- specifically as the link which takes a developed 
product* and sends or gives information .about the product (or 
distributes and delivers the product) to the User population,** Thus, 
on this conceptual basis, we picture as distinct a separate Dis- 
semination element of the R/D5I process--an element having tangential 
boundaries with Producers /Developers on the one hand and with Users 
on the other. 

In a more complete systems sense, however. Dissemination cannot 
be viewed as a basically isolated segment. Precisely because it is_ 
the link between Research/Development on the one hand and User 
Selection/Implementation/Utilization on the other, we must conclude 
that for both theoretical and practical purposes. Dissemination 
either does not exist or exists dysfunctional ly when the Research/ 
Development and/or the Selection/Implementation/Utilization functions 
occur inadequately or inappropriately, or (in the case of Selection/ 
Implementation/Utilization) do not occur at all. 



*As noted in earlier chapters, for simplicity of usage we use the term 
"products" whenever we are referring to the outputs of the Development 
process. H«owever, the reader should keep in mind that the tenn 
"products" is meant here to convey the full array of Development 
outputs — programs , proces ses , mode Is, s trategies , approaches , e tc . 
as well as the narrower range of outputs we typically think of as 
"products ." 

**We recognize that the definition of Dissemination that has achieved 
some degree of consensus in education today includes two-way communi- 
cation between Producers and Users, with Users "feeding forward" to 
Producers information about their perceived needs as well as evaluative 
feedback on the products they have tried to use. However, although we 
are aware that & combined Dissemination and Feed-Forward (D/FF) system 
is being designed by NIE, we view Feed-Forward as a conceptually discrete 
function, and therefore do not treat it here in this analysis. Feed- 
Forward is an additional function we might consider in some future, ex- 
panded analysis. 
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In a word: 

a. The kind of Dissemination system needed in a given sector 
is dependent on the nature and level of maturity of the 
total R/D5I system. 

b. A Dissemination system is useless (and even dysfunctional) 
if it is not designed rn accordance with the realities of 
the existing overall R/D5I system. 

By way of contrast, it is more possible to think of Research 
and Development as separate* isolated elements simply because there 
is an identifiable end result — there is a "product," even if it 
figuratively "sits on the shelf" and is never used. 

Similarly, it is also more possible to think of Implementation/ 
Utilization as separate, isolated elements. Users do implement and 
use products, methods, processes, etc. — even though few Users may 
use a particular product; or the product is of poor quality; or the 
product is used inappropriately or inadequately. Also, these products 
may or may not have come from external R§D-based sources (in a 
direct sense). On the other hand, the "end product" of Dissemination 
is the effective Sel ection/Implementation/Utilization by relevant 
Users of products which are both adequate and relevant to the User's 
need. 

Thus, it makes sense to think of the Development/Dissemination/ 
Implementation/Utilization process as a package with the focus being 
on the User. 

To illustrate, where the overall R/D^il system is immature, several 
conditions will tend to exist which significantly impact upon the 
effectiveness (and thus upon the design) of the Dissemination system. 
1. The transforms between the steps of the R/D5I process 
are not clear. That is, it is not obvious (for example) 
how Research results can be applied to Development; how 
quality control is to be done in relation to Production; 
how the product will get to the User. 
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2. User knowledge tends to be highly limited. The User 
may not know a product exists, or how to differentiate 
between a "good" and a "bad" product; or how to effec- 
tively use a product--or, because of these factors, 
the User may not even try the product. 
Thus, when the R/D5I system is immature, certain kinds of Dissemination 
policies, strategies and mechanisms may be needed which are not needed 
when the R/Dftl system is mature. 

Viewing the Development/Dissemination/Implementation/Utilization 
process as a "package," it now becomes possible to consider the 
question of appropriate Federal agency policies and strategies in 
Dissemination. As a preliminary comment, we may note that Agency 
options are limited by certain factors: 

1. The further the Dissemination "package" goes into the 
User system, the less control can or should be maintained 
by the Agency. That is, the Agency may have high control 
over the Dissemination aspect, less over User Imple- 
mentation, and even less over Utilization. 

2. The larger and more fragmented the User system, the more 
difficult it becomes for the Agency to be directly involved 
(in an effective manner) with Users. 

The discussion which follows is predicated upon the above under- 
standings of the Dissemination process and of Dissemination systems. 
V/e will first examine what role Dissemination has in the overall 
R/DSI system. From this context, we will be able to identify some of 
the key building blocks of a Dissemination system. Next, we will 
focus on strategic elements of Dissemination as a process and then 
on some of the likely key problem areas for Disseminatio We will 
also comment on the state of knowledge concerning Dissemination as a 
total, complex process. 

From the above, we will suggest some relevant Dissemination 
strategies. The outline and process of the discussion which follows 
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may be represented schematically as in Figure 1. 
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2. The Place of Dissemination in the R/DSI System 

To understand the nature of (and effectively develpp) a 
Dissemination system, it is vital to have a clear understanding of 
how Dissemination fits into and affects the overall R/D5I process. 

A. System-Creating Phenomenon 

In the first place, we must be aware that Dissemination is a 
system-creating phenomenon. 

This is true because without Dissemination a total R/D5I system 
simply does not really exist. Research and/or Development can 
indeed exist independently — but unless there is also a link to the 
Users, Research merely produces knowledge in isolation, and Development 
merely produces products which, in effect, ''sit on the shelf." Thus, 
mechanisms for disseminating the results of Research and Development 
to Users must exist for a total R/D5I system to exist. 

Inhere the R/D5I system is immature. Dissemination mechanisms may 
have to be created (where none exist) and/or supplemented and sup- 
ported (where inadequate mechanisms exist) . Where the R/D5I system 
is more mature and formalized, institutionalized Dissemination 
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arrangements may exist in the form of mechanisms that permit direct 
Producer (or Supplier)/User relations. 

B. A Linkage Process 

It follows from the above that Dissemination, in its essence, 
is a linkage process — a linkage process so important and so critical 
that the total R/D§I system does not exist in its absence. The 
functions of this linkage process are essentially three-fold: 

a. To inform Users of the results of Research and Development 
which are relevant to and usable by them. 

b. To enable Users to effectively utilize the results of 
Research and Development. 

c. To enable Developers and Producers to develop and produce 
products which fit User needs. 

Viewed from this perspective, several important implications become 
obvious: 

a. It is vital that the Dissemination systems operate with 
clear understandings of the User organizations with which 
they interact. 

b. Dissemination systems and mechanisms must be tailored to 
the nature and style of the User organizations. 

c. The products to be disseminated must be matched both to 
User needs , to User capabilities , and to User readiness 
to adopt new products, programs, processes, and even new 
Dissemination mechanisms. 

d. It is important to keep in mind that the relevance of any 
specific Dissemination mechanism is determined by whether 
or not it helps achieve Dissemination purposes. A specific 
Dissemination mechanism is not important in and of itself. 
Thus (as we shall later note), in any given situation, 
several different Dissemination mechanisms may be reasonably 
effective. 
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However, we must also note that normally most Users will tend to be 
slow to adapt to new information. sources, new mechanisms, new modes 
of operation, new materials and products. In this regard, it may 
at times be good strategy to utilize and improve an existing Dis- 
semination mechanism (with which the User is familiar and comfortable) 
rather than to introduce new Dissemination mechanisms (which in them- 
selves may require change by the User) . 

C. Optimizing Interaction and Fit 

Finally, from all of the above, it follows that in the design 
of Dissemination processes there is a very high premium on designing 
Dissemination mechanisms that will optimize interaction with Users 
and will thus also tend to optimize the fit between Products and 
Users. However, as a strong caution, we must also make a distinction 
between creating mechanisms for Dissemination and achieving Dissemina- 
tion objectives. That is to say, the mere existence of Dissemination 
mechanisms does not guarantee that Dissemination will actually occur. 

D. Many Products and Modes 

From the previous discussion af the importance of designing 
Dissemination mechanisms which optimize interaction with the User 
and fit between Product and User, an erroneous conclusion could 
easily be drawn — i.e., that there is one best set of Dissemination 
mechanisms and that NIE should discover and design that set. There 
are at least four important reasons why such a conclusion would not 
only be erroneous, but also potentially dysfunctional. 
1 . Many Products 

How a Dissemination system will function in practice 
depends to a significant degree on the types of products 
being disseminated. Even within a given sector (e.g., edu- 
cation or health), there will be a degree of product variation 
sufficient to call for certain differences in Dissemination 
mechanisms. For example, the methods most appropriate for 



Dissemination o£ a very simple product are likely not to be 
at all appropriate for the Dissemination of a very complex 
technology; and vice versa. Thus, there must be a fit between 
the "what" (Product) and the "How" (Dissemination) --a fit 
that will tend to vary across product types. 

We may also note here that this linking of product type 
with types of Dissemination mechanisms can, over time, lead 
to some degree of specialization by Disseminators in relation 
both to the type of products which they will tend to dissemin- 
ate and the Dissemination mechanisms which they will tend to 
utilize. 

2. Many Modes of Dissemination 

In any given sitiaation, there are likely to be several 
modes or mechanisms which would be reasonably and more or less 
equally effective for the dissemination of a particular product 
or type of product. In such a situation, the issue is not: 
Ifliich is the best? ; but rather: Which is available? V/hich 
already has User confidence (or lack of confidence)? What 
are the cost differentials? etc. 
5. User Familiarity and Acceptance 

It is important that Users have sufficient confidence in 
the source of Dissemination information so that they will be 
at least initially favorably inclined to try out an innovation. 
It is not unreasonable to anticipate that in many instances, 
out of all the possible relevant and reasonable Dissemination 
mechanisms, a User (or group of Users) will have confidence 
in one particular Dissemination mechanisms because of prior 
positive experiences with it. Thus, where possible, it would 
be reasonable to utilize that mechanism. Given the large number 
of Users nationwide, we would expect different User sets to have 
different "favorite" Dissemination mechanisms. Each separate 
Dissemination mechanism may gravitate towards a specialization 
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in the dissemination o£ a relatively limited line or type 
of product, but for that line or type of product, it is 
potentially a highly effective Dissemination tool for a 
given set of Users. This need not be viewed as a factor 
limiting improvement in practice. As we have noted, a wide 
variety of different innovations may be potentially helpful 
to any given User; and presenting a User with too wide of a 
set of alternatives may result in a dysfunctional state of 
confusion. 
4. Fail Safe 

As we will note more fully later, the Dissemination system 
needs to be made fail-safe, since it is reasonable to expect 
that Users will from time to time have "bad" experiences with 
Dissemination sources, which when going through a "learning 
curve" may lead to premature and long term rejection. 

We have just noted that when the User has had "good" 
experiences with a Dissemination source, the positive experience 
should be reinforced and capitalized upon. The reverse is true 
when the User has had "bad" experiences with a Dissemination 
source. In this situation, reinforcement should be avoided-- 
but if there is only one Dissemination source available to the 
User, the "bad" experience will be reinforced and the User 
will tend to discredit and reject the entire Dissemination 
system. 

Thus, it makes sense to provide the Users with several 
alternative Dissemination sources, such that if they experience 
failure with one source, they can go to another source instead 
of rejecting the entire Dissemination system. 
In summary, there are many Dissemination modes or mechanisms 
which could potenially optimize interaction with the User and the 
fit between products and Users. No single mode or mechanisms (or 
even set of mechanisms) is likely to be clearly 'the best." Further, 
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there are a number of very good reasons to follow a policy of per- 
mitting and supporting the successful functioning of a variety of 
Dissemination systems. 

E. R/DSI System iMaturity 

The type of Dissemination system required varies as a function 
of the degree of the maturity of the R/D5I system. 

1. Inmmture R/DSI System 

When the R/D5I system is immature, several key conditions 
will tend to exist: 

a) There will be general lack of quality and quality control 
in Development (and also in Research), 

b) Users will generally have a low level of effectiveness in 
Implementation; Users may be (to a large extent) unaware 
of what is available for their use; and Users will lack 
the evaluative capability to distinguish between "good" 
and "bad," "relevant" and "inappropriate" products. 
Under these conditions, the Dissemination system must be 

designed to find, evaluate, categorize, store, and retrieve 
information about what is available. Further, the Dissemination 
system must provide mechanisms to enable Users to be aware of, 
properly select, and be capable of implementing the particular 
Development product which is most relevant to and usable by them. 
Such mechanisms may be provided either by the cireation of new 
mechanisms or by the utilization, modification or improvement of 
existing mechanisms. 

2. Mature R/DSI System 

When the R/D5I system is more mature, a different set of 
conditions tends to exist. 

a) Products of good quality and mechanisms for quality control 
will be more readily available. 

b) Users will tend to have a higher degree of familiarity with 
products; they will better know how to find, select, imple- 
ment, and use new products. 
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Under these conditions, an agency with an R/D§I system-build- 
, ing mission will have les need to be concerned about estab- 

lishing Dissemination mechanisms (though there may still be such 
a need) and will be able to iocus its attention on different 
specific aspects of the overall Dissemination process (e.g., 
ensuring outreach and disseminating thin market products to small 
special groups) . 

F. The Role of Intermediary Organizations 

Tlie role of intermediary Dissemination organizations deserves 
special attention at this point. (By intermediaries we mean organiza- 
tions external to both Producers and Users.) 

As described above, the conditions existing when the R/D§I system 
is immature may require that intermediary Dissemination organizations 
and mechanisms exist to perform those functions which are not being 
performed (or are being inadequately performed) by Producers/ 
Developers and Users (e.g., quality control. User search, testing 
evaluation). Indeed, it is likely that the Agency will have to estab- 
lish such intermediary mechanisms. 

However, as the R/D5I system matures, the need for intermediaries 
may decrease over time and the role of intermediaries will change. 
For example, as Producers become more capable of developing quality 
products, the need for intermediary quality control mechanisms diminishes. 
Indeed, the* need for some intermediary mechanisms and functions may 
completely cease to exist. 

A key issue arises for the Agency in connection with these inter- 
mediary mechanisms. Organizations and institutions, once established, 
tend to seek to continue their existence. Thus it becomes imperative, 
from the beginning, that the Agency build adaptability into the 
intermediary mechanisms it stimulates and funds — and even provide for 
the possible demise of these mechanisms. 

G. Long Term Objective: Developing a Mature System 

A basic long-term objective of the Agency should be to facilitate 
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the transition of an immature R/D§I system into a more mature system. 
The Dissemination system should be seen as a key element in achieving 
this objective for two reasons:* 

a. The vital role of Dissemination as a linkage process 
requires that mature Dissemination mechanisms be a part 
of any mature R/D§I system. 

b. Under the conditions existing in an immature R/D§I system, 
the intermediary Dissemination mechanisms tend to fulfil 
functions usually associated with Producers and Users in 
more mature systems. Thus, Dissemination becomes a 
potentially strong entry point for facilitating the matura- 
tion of other parts of the total R/D§I system. 

3. Key Building Blocks of a Dissemination System 

From an understanding of the overall R/D§I system and of the place 
of Dissemination in that system, it now becomes possible to identify 
some key elements (cr building blocks) of a Dissemination system. 

A. Elements which are Primarily External to the User 

1. Information from Knowledge Sources 

Mechanisms must be provided that can determine what is 
available. This could be information about products 
currently under development; products that have been around 
for a long time but have not been widely used; or even 
exemplary practices within User organizations which have 
not become known outside of a particular User organization. 

2. Quality Control 

Mechanisms must be provided to sort out the "good" from 
the "bad" products, i.e., those products which are useful 
and relevant as contrasted to those products which do not 
work or are not appropriate. 

3. Sorting and Retrieval 

As there will likely be more products than a User can 
reasonably be expected to be aware of, some mechanism must 
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be developed which will allow Users to identify and obtain 
the particular products which are relevant to their needs, 

4, Tailoring 

Because of the diversity among User organizations and their 
needs, it is probable that existing products will not quite *'fit" 
User needs and will thus have to be "tailored" to fit a par- 
ticular User organization. 

The relative definitiveness of the problem/solution fit in 
a given instance would significantly influence how these differ- 
ent elements might be used in working with a given User, In 
some instances, there might be a variety of innovations that 
could assist Users in their (often loosely defined) needs, and 
one might decide to provide them with information on a broad 
array of alternatives. In other cases, however, it might be 
far from clear which products would be more or less useful. In 
these circumstances, it becomes the objective of the process to 
eliminate the clearly inappropriate, while narrowing down the 
choice to roughly comparable alternatives from which the User 
could select, 

B, Elements which are Primarily in the Domain of the User 

1, Trial 

In the initial stages of Dissemination, whether initiated 
by the User, by Producers/Developers, or by an intermediary 
agency, the nature and method of Dissemination must be 
designed so as to motivate the User to give the new product 
a trial. Unless this is done, the overall Dissemination 
process breaks dovm and stops, 

2. Adoption and Implementation 

Dissemination is not merely a "sending out" of information. 
Dissemination must also be concerned with what is done with 
the information after it is received and tried by the User, 
This should be obvious, for the purpose of "sending out" 
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information is to increase the Adoption and Implementation 
of new products, materials, processes, etc. The point is 
emphasized here because Dissemination often is understood 
simply as a "sending out," with a resulting neglect of what 
Users do — or are able to do — with the information they 
receive, f 
3, Utilization 

Dissemination systems must also include provision for 
continued Utilization — not just Adoption and Implementation 
followed by discontinuance. 

The role of Dissemination in Adoption, Implementation, and 
Utilization is to insure that Users will receive the help they 
need to make effective use of what they receive, 

C, Elements which may be Either Producer or User Initiated , 

1, User Awareness 

Mechanisms must exist through which the User can become 
aware that a product exists which may be relevant to his 
needs . 

2, Matching to User Needs 

Mechanisms must exist by which the sources and/or Sup- 
plier of products and information on products can become 
aware of User needs in order to permit the sorting/retrieval 
and tailoring described above. 

In both of the above, the initiative may come validly from 
the Users, from Producers/Developers, or from intermediary 
agencies. The Users may be looking for a particular innovation, 
or a message may be sent to the Users to inform them of a par- 
ticular innovation, 

4, Devising Producer/User Linkage Strategies 

A, Dimensions (Sub-Blocks) of Dissemination System Building Blocks 
Each of the building blocks considered above miglit be described 



in terms of dimensions (or "sub-blocks")* Thus the building 
block "information from knowledge sources" might be elaborated 
by reference to such sub-blocks as: the distribution of printed 
materials, face-to-face presentations, conferences, demonstration 
programs, etc. 

The design of Dissemination strategies is essentially a 
process of combining these building blocks, and especially the 
various building block dimensions or "sub-blocks". The effec- 
tiveness of these strategies is likely to be dependent on the 
extent to which these component building blocks and "sub-blocks" 
are properly selected, sequenced and timed for a particular 
target User and a particular innovation. For example, in one 
situation an appropriate Dissemination strategy might consist of: 
a series of introductory "flyers" or brochures; a visit to a 
User to ascertain needs; search and sorting to identify appro- 
priate available products; a second visit by a representative of 
the disseminating agency; perhaps followed by informal discussion 
to zero in on a given innovation; a demonstration by other current 
Users of the innovation; and a promise of needed service and 
support. In a different situation, such a Dissemination strategy 
might be inappropriate. 

B. Key Issues in the Design of Dissemination Strategies 

From the above we can identify at least three key issues 
that must be considered in the design of Dissemination strategies: 

1. IVhat is the appropriate combination of "building blocks" in 
a given situation? 

2. What is the appropriate organization for implementing the 
Dissemination strategy in a particular situation? In determining 
the appropriate organization, some relevant questions might be: 
Who works with the User? IVho has the relevant knowledge and 
skills? IVho knows (or can obtain sufficient knowledge about) 

the particular innovation? 



145 



3* Vfliat type of role is the Agency to have. in implementing a 
Dissemination strategy? For example, will it have a direct 
operational role in the' Dissemination process, or will it 
play an indirect role facilitating the activities of Producers, 
Intermediaries, and Users? In either case, what will be its 
major objectives and modes of operation? 

5. Factors Affecting Dissemination as a Process : 

A. Cost/Effectiveness. as a Factor 

At the very outset, one must be aware that Dissemination is a 
very cost-dependent process. The Disseminator has large publics at 
both ends of the process. That is to say, the Disseminator must be 
in touch with the various Producers and Developers, on the one hand, 
and with the various Users, on the other. Further, there are a variety 
of Dissemination media to be considered,, each with different costs 
attached. 

Much of the discussion about Dissemination in the literatiore 
focuses on the effectiveness factor, but less emphasis is given to 
cost/effectiveness. This is unfortunate, for the Disseminator must 
consider cost/effectiveness of various Dissemination media in relation 
to the dissemination of a particular innovation to a particular User 
public. For example, mailing 10,000 brochures may result in a low 
percentage of response (in terms of Adoption, Implementation, Utiliza- 
tion) , but such a mailing might be relatively inexpensive and could 
reach a very sizeable audience. Hence, even with a low response rate, 
a substantial impact might be attainable for the cost and time involved. 
By way of contrast, having three staff members make trips to User sites 
could tend to be more effective (with the Users contacted) but might 
be relatively more costly and time-consuming and would reach far fewer 
Users. Thus, cost/effectiveness considerations would be a key factor 
in choosing. between~thes"e ~tw^^ illustrative Ttrategies . 

We must further note that in most instances, it will not be 
obvious (or even determinable in advanced) what strategy is in fact 
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'•best." All sorts of strategy combinations will succeed to some degree 
Thus, the Disseminator is not working in an area where he can say there 
is ''one right way." C<^^ course, there may be clearly identifiable 
"wrong ways" of Dissemination relative to a particular innovation and 
a particular User public, and identifying these "wrong ways" may be an 
important issue for the Disseminator.) 

Most likely, an optimal strategy will be to include a variety 
of approaches; and the specific combination of approaches that should 
be used will be determined by such factors as: objectives; number and 
nature of Users; cost; availability of Dissemination resources (e.g., 
materials, skilled personnel, support equipment such as audio-visual 
equipment, e tc . ) ; etc . 

In addition, as we have said, there may be much uncertainty as 
to the benefits derivable from alternative products. Since products 
may vary in cost to the User, and there is often also an interaction 
between the nature of the product and how (and hence at what co.^ • 
it can be disseminated, there can be both considerable variation cL-i^d 
considerable uncertainty associated with overall cost/effectiveness 
considerations (where cost is now seen in both User and Disseminator 
terms) . 

B. Professionalism and Reliability Factors 

IVhatever approaches the Disseminator decides to use, it is vital 
that these be reliable and be used in a professional manner. Sloppy 
brochures, publications that are not delivered when promised, products 
and programs that are inappropriate, unclear or unusable— when 
unprofessional and unreliable Dissemination efforts such as these occur 
the immediate and long-term effects can be disastrous for the overall 
Dissemination process. In the short term, the result is likely to be 
that the User will give little or no attention to the innovation. In 
the long term, the result is likely to be that the User will reject 
out-of-hand any future Dissemination efforts by the Disseminator-- 
whatever the Dissemination strategy. 
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The basic reason for this dynamic is that the User lives in an 
operational framework. In contrast to the Researcher or the Developer, 
the User cannot afford the luxury of an iterative process of experi- 
ment/ ifflprove/experimeqt/i2«p7:3ve/etc. 

C. Motivation m<l Commitment 

From the above, it is obvious that the Dissemination process 
requires a great deal of hard work. Further, the process is subject 
to a great deal of potential discouragement on the part of the Dis- 
seminator. Hence, Dissemination needs to be done by persons and 
organizations vAio feel a degree of "excitement" about their task--i.e., 
those who are interested in and committed to the particular . innovation 
and/or to improvement in the sector as a whole. 

D. Trust 

An effective Dissemination process is based on trust--trust in 
the information being disseminated, and trust in the' source of Dis- 
semination. Quality control is a key to building and maintaining 
trust — quality control on the products and quality control on the 
usage of the products. Simply put, the Disseminator should not be 
promoting a product under conditions that will lead to its misuse 
(e.g., under conditions where the User does not know how to use it; 
or where the product is inappropriate to the User's need). 

We may further note that interpersonal mechanisms tend to be seen 
as more trustworthy than printed and other impersonal mechanisms. This 
is especially true in areas of innovation where there is a low level of 
development and a high level of uncertainty. 

E. Replicability of Developed Dissemination Media 

When considering any particular element of a Dissemination 
strategy, it is important to make a conceptual distinction between 
the design stage of devising the strategy and the subsequent operational 
stage of using it to achieve the Dissemination objectives. The design 
stage is likely to take considerable time--deciding what to do, develop- 
ing it, making it usable through pilot testing, evaluating, refining, etc. 

148 



Once the element of Dissemination is ready for use, however, it 
can generally be reproduced rather rapidly. For instance, the develop- 
ment of the next in a series of brochures is usually a short-term 
effort. Similarly, training of "detail men" once the pattern has been 
established tends also to be a relatively short-term effort. 

This characteristic of replicability can be utilized in order to 
expand Dissemination programs rapidly and efficiently. However, in so 
doing, it is important not to follow the tendency to overuse particular 
dissemination media simply because it can be done more cheaply and 
easily. 

F. Overall: A Complex Process 

Though at times it may appear deceptively simple. Dissemination 
is a complex process. There will be several steps' in the process, 
and these are likely to be interactive. For example, poor quality 
control which results in a "bad product" being exposed to a User may 
negate the best Dissemination strategies. Similarly, unprofessional 
and unreliable Dissemination efforts will likely result in reduced 
feedback from the User (relative to the innovation), thus reducing the 
User information needed by Researchers and Developers. 

Thus, while an individual element of the Dissemination process may 
be relatively simple and manageable, the overall process is complex. 
There are many different steps in the process. There are many different 
kinds of Users, Developers, Producers, Intermediaries, Products, etc. 

It becomes obvious,' then, that designing and managing the Dis- 
semination system requires considerable skill, and this becomes a key 
issue for the Agency funding and building such a system. 

6. Key Problem Areas^ 

From an understanding of the R/D5I process, it is possible to 
delineate several key generic problems which must be dealt with by the 
Disseminator if Dissemination is to be effective. 
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A, Scope of the Problem 

1, Size of the User Population 

In amy sector, there may be a large number of potential Users 
of innovations. In the education sector in particular, there 
may be potentially thousands of Users nationwide. Just making 
effective contact with so many Users is likely to be a signifi- 
cant problem in itself, 

2. User (Market) Variations 

Within any large User population, it is inevitable that there 
will be variations among Users. There may even be variations 
across types of User sub-groups. Further, variations may exist 
across different facets of User populations and sub-groups. For 
example. Users may vary according to: needs; location; size; 
private vs, public status; profit vs. non-profit status; capa- 
bility to acquire, test, evaluate, implement and utilize innova- 
tions; etc. In the education sector in particular, the variations 
are likely to be quite numerous. Given such variations, several 
specific problems may arise. For example: 

a. It may be difficult to identify which Users are the relevant 
potential target population for Dissemination of a particular 
innovation, 

b. The needs of the various Users are often hard to identify and 
define. 

c. Market segme'ntation may be necessary. 

B, Producer/User Breakdowns 

A second key area for potential problems in Dissemination is the 
relationship between Producers and Users, We can specify several 
conditions under which the direct Producer/User relationship is likely 
to break down, 

1 . Information Flow 

Obviously, the flow of information between Producers and Users 

is a key element in the Dissemination process. However, this 

ERIC 



flow o£ information may be hampered under the following condi- 
tions : 

a. There are language differences between Producers and Users 
(e.g., the use of R5D vs. operational language and concepts). 
For another example, in certain education situations, key 
Users may be Spanish-speaking, while the Producers are 
likely to be English-speaking. 

b. Tliere is a lack of adequate communication channels between 
Producers and Users. This is especially likely to be the 
case when the R/D5I system is immature or where the market 
(Users) is fragmented and diffuse. 

c. Users lack the capability to understand the technical 
aspects of the innovation. Here, even when the information 
disseminated is technically adequate and reaches .the relevant 
Users, they may be incapable of using the information received 

2. Producer Motivation 

For a variety of reasons. Producers may have a low level of 
motivation to disseminate new products, especially in the face 
of resistance, uncertainty and/ or fragmented markets. 

a. Some Producers may think of themselves simply as Development 
organizations and may have little interest in carrying out 
the Dissemination function. In these instances, their 
organizations may have a low level of Dissemination experience 
and skill, a fact which will also tend to lessen the motiva- 
tion to disseminate. 

b. Under conditions of perceived resistance, uncertainty or 
fragmented markets, even those Producer organizations which 
are oriented towards Dissemination and which therefore have 
the necessary Dissemination skills may decide that the cost 
and risk are too high to attempt a high level of Dissemination 
An example would be school textbook publishers who might be 
very hesitant to produce and promote innovative materials 
which have not yet been "accepted" by Users --or which perhaps 
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have a limited (thin) market. 

3. User Motivation 

For effective Dissemination to occur between Producers and 
Users, the Users must be motivated to receive, test, try and 
evaluate innovations from Producers. A number of factors act 
as potential barriers to User receptivity. 

a. Being exposed to an innovative product can raise the 
posssibility (to Users) that they are not currently 
"doing it right," or at least not doing something as well 
as it might be done. This is not always an easy thing for 
a User to admit. 

b. To study, test, evaluate and then implement and utilize an 
innovation takes time and is therefore an interruption in 
the work of Users who may already feel overwhelmed by demands 
on their time and energy. Where training is required for 
effective usage, additional demands are placed on the time 
and energy of the User. 

c. The above discussion would seem to imply that Users generally 
are confronted with a small number of innovations. In many 
instances, it is more likely that the User will- be confronted 
with a plethora of innovations. When this is the case, the 
User is likely to feel overwhelmed by a seemingly impossible 
task of reviewing, testing, evaluating, etc. — and may thus 

be inclined to ignore everything offered. 

A somewhat different issue is involved in those cases where 
an innovation has been developed by a User. In this case the Dissemina- 
tion and the completion of the Development process become interwoven. 
The innovation has to be' identified, possibly modified and packaged for 
a wider User group and then disseminated. A key problem is that there 
is generally little to motivate a User/ Developer to become involved in 
this process. J 
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4. Skills 

a. Producer Dissemination Skills: 

. Dissemination requires special skills, which may differ 
according to the type of User and/or the type of innovation. 
Thus, the Producer/User relationship may break down where: 

1) the Producer is inexperienced in Dissemination ; 

2) the Producer is inexperienced in Dissemination in a 
particular market or in relation to a particular type 
of innovation . 

b. User Implementation/Utilization Skills: 

The Producer/User relationship may also break down when 
the User lacks the skills necessary for Implementation and 
Utilization. It is not enough to assume that a "good" 
product can be effectively used by a User. Training may be 
required. 

c. User Dissemination Skills: 

As before, where the User is also the Developer there may 
be a lack of skills in. knowing how to select innovations for 
Dissemination, how to tailor for general applicability, and 
how to communicate to relevant others. 

5. Supplemental Resources 

The Producer/User relationships may also break down, when 
effective Inplementation and Utilization require supplemental 
resources which the Producer does not provide and/or the User 
does not have (and may not be able to acquire and/ or utilize). 

6. Patterns of Adoption 

The rate of Usier acceptance and absorption of new information 
and new information sources tends to be slow^ What is ijnportant 
to note here is that the rate of User absorption of new information 
sources — his awareness of the existence of such sources, his 
interest in making use of these sources, his trust in these sources, 
etc. — tends to be much slower than the rate at which these informa- 
tion systems can be developed and made active. 
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As a result, there is the danger that creating large 
Dissemination systems may lead to high expectations (by both 
the Disseminators and the Users), while in reality the benefits 
and satisfaction may be strongly limited by the slow rates of 
absorption and utilization. 

In a word, the capacity to develop and activate Dissemination 
mechanisms tends to be greater than User capacity to absorb and 
utilize the information sent to them by the Dissemination mechan- 
isms. Thus, in designing and creating Dissemination mechanisms, 
the Agency needs to assess carefully its expectations in terms 
both of the expected levels and the anticipated time-lines of 
User Acquisition, Implementation, and Utilization. This assess- 
ment also needs to be made by the personnel of the newly-created 
Dissemination mechanisms. 

C. Knowledge about the Dissemination Process 

The discussion thus far has focused on various elements of the 
Dissemination process. Now it is imjjprtant to note that while much may 
be known about specific elements of*^he Dissemination process (e.g., 
about the impact of written as compared to interpersonal communication) , 
relatively little is known about Dissemination as a complex total 
process. We must further note that merely "adding up" our knowledge 
of individual elements will not provide a useful understanding of the 
Dissemination process, for these parts are interactive and influence 
and change each other and the overall process. 

Given the uncertainty associated with the workings of the total 
Dissemination process, there is validity in supporting processes of 
natural development which would lead to the use of mixed and variable 
strategies in a Dissemination system design. This contrasts with a 
strategy of seeking optimal designs at this point in time. 

7. Design Requirements and Strategies 

Up to this point we have been developing an understanding of the 
R/D5I process . The importance in so doing is to increase our ability 
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to design effective Dissemination systems, develop effective Dissem- 
ination strategies, and thereby, to recognize the appropriate Agency 
role. It is to this task that we now turn our attention. 

A. Congruency and Adaptability 

A Dissemination system must be congruent with the state of develop- 
ment of the R/D5I system it serves. What is needed (and effective) for 
an immature R/D§I system may not be needed (or effective) in a mature 
system. Indeed, the need for some aspects of the system may cease 
to exist. 

Design Requirements : 

The Dissemination system design must be flexible and 
adaptive over time, and even provide for the termination 
of some of its parts. 
Strategy : 

Institutions tend to be slow to change and even to 
resist change. Certainly, it is not typical of an in- 
stitution to seek its own demise. Thus, an important 
strategy for a funding agency is to build adaptability 
(and even termination) into the initial design of the 
overall Dissemination system. Dissemination sub-systems 
(e.g., regional D/FF systems), and Dissemination mechan- 
isms. 

B. Stability 

Though institutionalized Dissemination mechanisms must be adaptive, 
they must also have stability in order both to avoid dysfunctional dis- 
ruptions and to benefit from the confidence and competency which comes 
from experience and familiarity. 

When change is abrupt and continual, a number of dysfunctional 
results tend to occur: 

1. Dissemination agents do not have time to accomplish their 
objectives . 
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2. Dissemination agents will tend to become uncertain, 
confused, insecure. 

3. Dissemination agents vdll not gain the confidence and 
competence that comes from experience and familiarity. 

4. Dissemination agents will not have credibility with Users. 

5. Users will become confused and will tend to resist further 
Dissemination efforts. 

6. The Disseminator/User relationship will lack continuity. 

7. Communication channels will be inadequate and tend to be 
unknoivn to the User . 

Design Requirement ; 

The Dissemination system design must provide stability 
and gradiialness to change for the system as* a whole and 
for its parts. 
Strategy: 

Stability and gradualness of change should both be a 
part of the initial system design and an integral part of 
the funding support and programming of the Agency. 

C. Janagament Mode 

The nature of the control mechanisms u^^ed to manage a system 
is likely to have a significant effect on the nature and effectiveness 
of the system. Further, the appropriateness of different management 
modes depends to a large degree upon the level or state of the system's 
maturity. 

Design Requirements : 

1. The mode of management control vrhich is relevant to 
the current level of maturity of the overall R/D5I system 
must be built into the Dissemination system design. 

2. Provisions must be made in the Dissemination system 
design for the management mode to change over time as the 
overall R/DSI system matures- 



156 



3. Information feedback mechanisms must be an essential 
element of the Dissemination system design to provide 
system managers with indicators of the changing maturity 
level of the system, and to signal the need for change 
in the mode of management control used. 
Strategies : 

In a mature R/D§I system, management of the Dissemination 
functions normally can be done through the development, use 
and control of well-developed plans. However, in an immature 
R/D§I system, this mode of management is not so feasible be- 
cause there is too much uncertainty, too many variations, etc. 
Rather, a relevant management mode is one which focuses on 
policy rather than detailed administrative management of plans 
a mode of management which steers and guides the various parts 
of the system; a mode of management which oversees the pro- 
cess of system development and change. 

In this management mode, monitoring of the process becomes 
they key management tool and a key Agency role whether per- 
formed directly or contracted out. 

D. Quality Control 

As was noted earlier, the quality of innovative products is 
important to an effective Dissemination process, l^hen product quality 
is low. User trust in the Dissemination soiorce deteriorates. 

Design Requirement : 

Mechanisms for quality control of Development/Production 
; outputs to be disseminated must be an integral part of the 
•^Di"sse.Biination system design. 

Strategies: 

The Agency role here would be to monitor the quality 
control process and to intervene to create appropriate mech- 
anisms and processes where these were lacking. 
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E> The Product/User Match 

Regardless of the quality of a given product, it will not be 
of much value to Users if it is not relevant to their needs or if they 
are not capable of using it. 

Design Requirement: 

The Dissemination system design must provide mechanisms 
through which products may be adequately matched with User 
needs and capabilities. 
Strategies : 

Such matching can be accomplished in a variety of ways. 
For example: 

1. proper identification of products relevant to User 
need and capability; 

2. tailoring innovations to User need and capability; 

3. helping the User obtain and develop the capability 
to implement and utilize innovations. 

F. Buying/Renting Services 

It is rrot always necessary or wise that every component of 
a Dissemination system -be contained within a single Dissemination 

organization. Services can be purchased or rented, either tempor- 

arily or regularly. For example, successful Dissemination does not 
depend upon the Dissemination agent doing its own printing. Whether 
it is more cost/effective to have a printing capability within an 
organization or to purchase printing services is a valid administrative 
question, but it is not an issue vital to successful Dissemination. 

Design Requirement : 

Ths Diss^iniTiS-tion syst^ni d^^sig^ mimt be such as to 

allow a funding agency to treat the "buy/rent" question 

as an administrative issue. 

Strategies : 

The buy-or-rent decision may need to be orchestrated and/ 
or mandated by the Agency. 
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Product Championship 

There is evfidence from a variety of sources that having a "product 
champion" is a key factor in increasing the probability of successful 
Dissemination, The product champion may be a User, a Producer or 
Developer, an influential individual or group, or even the Dissemina- 
ting agency. Who the product champion might be can vary. What is 
in5)ortant is that the product champion be excited and committed enough 
to follow through on the Dissemination efforts. We do however note 
that prior negative User experience with product chaii5)ions can be a 
significant barrier to effective utilization of this strategy. 

Design Requirement : 

None. This is an administrative consideration. The 

design should neither mandate nor prevent the use of 

"Product Chanqpions". 

Strategy : 

For an Agency to become a "champion" may be dangerous. 
Its role is to ensure that the conditions (incentives, mechan- 
isms) for championship exist in the field. 

H. ■ Stimulating User Demand 

The likelihood that an innovation will be tried, adopted, im- 
plemented, and utilized is significantly increased when there is User 
interest in and demand for the innovation. Similarly, when a relevant 
product does not already exist. User interest and demand can be very 
effective in stimulating the development of a particular innovation. 
Design Requirement : 

None, This is an administrative consideration. The 

- • ^. 

effect • 
Strategy; 

It is often an effective strategy for a Dissemination 
agent to create conditions which will stimulate User interest 
in and" demand for innovation. When this strategy is utilized, 
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it is important that a funding agency control the rate of 
such "priming" so as not to permit stimulation of User 
demands which cannot be satisfied, or cannot be satisfied 
within a reasonable time frame. 

I. The Developmental/Emergent/Experimental Strategy 

Since the state of knowledge about Dissemination (as a total 
complex process) is low, there is justification for pursuing a policy 
which permits a degree of controlled, natural variation and provides 
mechanisms for monitoring of natural field experiments. 

The justification for such a strategy is three-fold: 

1. In conditions of immaturity and uncertainty, it is not ob- 
vious which specific mechanisms (or combinations of mechanisms) 
are most cost/effective (though certainly some building 

block elements of strategy are more effective than others 
for certain purposes) . 

2. Persons in the field may have valid insights about effective 
Dissemination methods. 

3. By using such a strategy, understanding of the Dissemination 
process in general (and within a specific sector in parti- 
cular) can be increased and, over time. Dissemination im- 
proved. 

Design Requirement : 

The design must provide mechanisms through which natural 
field experiments can be monitored and controlled. 
Strategies : 

1. A funding agency can establish a policy which provides 
support for natural experiments emerging from activities 
in the field. When tnis policy rs uniiz.e^ il ihuSt ub 
accompanied by monitoring and reinforcement. 

2. Monitoring is important both to control the process and 
to learn from the process. Reinforcement is important 
because there is a premium, on doing some things well--excellent 
ly--as compared to trying to do many things. To be avoided 




are failures due to p ;or operations which Users then blame 
on the overall Dissemination system, which in turn produces 
long-term negative systemic effects. Thus- when a partic- 
ular strategy (or element) is seen to work well, and the 
capability for perfoimng that strategy has been (or is 
capable of being) developed, then this strategy should be 
reinforced. 

3. What is to be avoided is the obvious mismatch of strate- 
gies to situations. 

J, Fail-Safe Mechanisms 

In spite of the premium on reinforcing successes and avoiding 
failures, failures will inevitably occur. There are just too many 
points of uncertainty and unreliability in the chain connecting R§D 
to Utilization which in combination result in low success probabilities. 
Further, when the^ R/D§I system immature, the probability of failure 
becomes quite high. Thus, when the R/D5I system is immature, it be- 
comes imperative that the Dissemination system be designed to be fail- 
safe. That is to say, if the User experiences failure in one instance, 
he will be aware that other alternatives are available— as contrasted 
to the User seeing the Dissemination system as a monolithic system, 
wherein the whole system is deemed useless by the User when he ex- 
periences failure with one part of it. 

, Design Requirement ; 

The Dissemination system must be designed so as to 

provide alternative channels of dissemination to the User. 

Strategies: 

Strategies should put a premium on redundancy, on making 
competitive alternatives available to the User. Such 
redundancy can be achieved in either ol two ways: 
1. Natural Decentralized Variation and Adaptation: When 
natural decentralized variation and adaptation are allowed, 
a variety of alternatives (even redundancy) may become 

161 



available to the User, and the User is thus less likely 
to transfer his perception of weakness in one part of the 
Dissemination system to the other parts. The various 
®^ Dissemination mechanisms should, of course, be orchestrated 
from a higher level, but with a minimum of visibility. 
2. A Fail-Safe Centralized System Design: While a fail- 
safe centralized system design is at least theoretically 
possible, it is complex and thus very difficult to develop 
and manage. If attempted, it would include: 

a. disaggregated parts; 

b. built-in competition among parts; 

c. built-in rediondancy of a few things done well. 

II, DISSEMINATION IN THE EDUCATIONAL CONTEXT 

1^ Level of Maturity of Educational R/DSI as a Whole 

It has been our contention throughout that we must understand an 
R/D5I system as a whole system f"^^ -^rder to develop relevant policies 
and strategies for any one aspect vf the total system. For example, 
many Dissemination roles and mechanisms which are created for an 
immature R/D5I system must change over time or even be terminated as 
the overall R/DSI system reaches maturity. 

In education, we find a very immature R/D5I system. There is a 
history of poor quality products, which suggests the need for internal 
mechanisms for quality control. There is a history of low levels of 
User capability in locating and utilizing products, even in defining 
problems and identifying needs. This history points to the need for 
intermediate mechanisms to facilitate and develop User capabilities. 

Thus, the immature nature of the overall educational R/D5I system 
strongly points toward the need for intermediate roles, mechanisms and 
linkages institutions in the educational Dissemination process. However, 
an understanding of R/DJII systems in general also makes it clear that 
in designing a Dissemination system for an immature R/D^l system, the 
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design must provide for adaptability, change and even termination of 
certain Dissemination roles and mechanisms. The importance of designing 
change into the system is based on the two previoxisly mentioned con- 
flicting considerations. 

1. Some Dissemination roles and mechanisms will outlive their 

^ relevance and usefulness over time as the total R/D^I system 
matures . 

2. Institutions tend to strive to maintain (and even enlarge)' 
their roles and their existence. 

Since such potentially desired "withering away" will not tend to 
occur naturally (or easily), the mandating of design for change (e.g., 
timed funding) and the necessary monitoring become key Agency roles. 

2. User Setting 

If workable Dissemination strategies are to be developed and 
effective Dissemination systems to be designed, problems inherent in 
two characteristics of the User system must be taken into account: 
a) its enormou:^ scale, and b) the attitudes toward change likely to 
be. encountered in User settings . 

A, Scale 

1 , An Enormous Number of Users 

There are more than 19,000 LEA's in this country. Each 
one includes many schools (literally hundreds of schools in 
the larger cities) . In each school there are numerous teachers 
not to mention students (who are in a real sense the ultimate 
Users). Thus, any idea about designing a Dissemination 
system to reach an entire universe of potential Users is simply 
mind-boggling, especially given limited federal resources 
allocated to the education sector. 

2. A Tremendous Diversity and Variety Among Users 

As might be expected with such an enormous number of 
Users, there is a tremendous amount of diversity and variety 
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among Users. There are: rich and poor Districts; urban, 
suburban, and rural Districts; public and private educational 
systems; large schools and small schools; Districts with a 
high level of teacher professionalism and Districts with a 
much lower level of teacher professionalism; etc, — not to 
mention variations in educational philosophy and in the nature 
and extent of administrative leadership. 

From the product perspective, there are many kinds of prod- 
ucts for which there is a relatively thin (and not commercially 
profitable) market — a fact which may point to a need to develop 
special mechanisms for disseminating- thin market products. 

Given the problems of scale, it would seem reasonable in developing 
Dissemination strategies and mechanisms to: a) identify the categories 
of personnel in different kinds of User settings who have the most impact 
on Adoption decisions, and b) focus Dissemination strategies and re- 
sources on these target Users. One possibility might be to differentiate 
LEAs in terms of the professionalism of their instructional personnel. 
The educational change literature is rather inconsistent in assessing 
the relative influence of teachers vs. principals and other administra- 
tors on innovation Adoption decisions ( a matter rather different from 

innovation Implementation). Some analysts enTphasize^ the , need,, for -par- 

ticipative decisionmaking on the Selection and Adoption of new programs, 
products, etc., with a major (perhaps the major) role for teachers. Other 
analysts urge strong administrative leadership as a preferred alternative, 
describing the real influence of teachers on such decisions as minimal 
and over-rated — and seem to suggest that this minimal role for teachers 
is to be viewed as desirable if innovation is to occur. What seems 
missing Trbm these analyses 1 3ttch factors as th^ 

professinalism of a District's teaching staff as a determinant of the 
appropriateness of one or the other of these approaches. Thus, for 
those Districts where the instructional staff demonstrates a high level 
of professionalism, the appropriate Dissemination strategy would seem 
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to be to target Dissemination resources directly at teachers. In those 
Districts where the instructional staff demonstrates a considerably 
lesser degree of professionalism, the target for Dissemination could be 
Superintendents, principals, and other administrators — on the assumption 
that the teachers are likely to ''go along with" the administrator's 
decision. Given the limited resources available for Dissemination, it 
might be wise to further segment the Districts with relatively low 
teacher professionalism into those characterized b) strong vs. weak 
administrative leadership and more vs. less interest: in innovation a: d 
work primarily with those Districts demonstrating interest in innovation 
and strong administrative leadership to promote and support innovation. 

At times, effective Dissemination may require a strategy of selec- 
tivity in the choice of User target groups with which to work. For 
example, in a diffusion model of Dissemination, the Disseminator might- 
work with only a few selected User groups. To consider another example, 
given limited funding, it may be best not to try to work with a resisting 
SEA. However, there is a danger that this kind of choice will be made 
simply on the basis of convenience. Thus, it may be necessary and 
appropriate for an Agency to mandate allocations of effort and to make 
available to Disseminators information on preferred strategies. 

B. Motivation and Technical Capability 

Resistance to innovation is probably the single most salient 
characteristic of User settings described in the change literature — 
not only in education but in other sectors as weJJ. In any given 
instance, such resistance may be attributable to attitudes, norms, 
and User system constraints; or to technical complexities and diffi- 
culties that make effective Implementation beyond the capabilities 
of User personnel in the absence of Implementation supports that 
are not provided; or both (as when awareness of the technical com- 
plexities generate resistant attitudes) . 

The attitudinal basis for resistance is simply that change tends 
to be threatening to people. In the education sector, the '^threat'* 
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aspect of change is probably even more significant because edu- 
cational innovation involves people change : new behaviors; philo- 
sophy change; technical strategies which require teachers to 
unlearn "old" behaviors and strategies with which they are familiar 
etc. 

However, there is suggestive evidence that the attitudinal 
dimension may well have been given excessive stress in the litera- 
ture and that the' technical problems may in the long run have far 
greater significance for determining the fate of innovations than 
the attitudinal ones. More often than not, teachers appear to be 
willing to give an innovative idea or program a try. But very 
r'ten, the programs fail because no one has given the teacher 
needed technical support--the kind of Implementation support 
and technical assistance that is required to make an innovation 
suc<:essful. The result is failure--failures which make it diffi- 
cult to introduce the next innovatior into that particular school 
system. Thus the ensuring of proper quality control and technical 
support components in Dissemination programs would seem to be 
of central importance- 
There has been increasing recognition of the importance of 
the technical side of resistance. Thus we find Dissemination and 
Utilization being clustered or integrated in recent educational 
policy. There is validity to this clustering of the Dissemination 
and Utilization functions. That is to say, it is not enough just 
to disseminate information and make Users "aware" of an innovation. 
The User must also have, be able to obtain, or be provided, 
Implementation/Utilization support throughout the Adoption/Installa- 
tion process in order to make sure that there is the best possible 
chance for the innovation to be successful, OlheiMise, the proba- 
bility of Dissemination failure is very large-- -with the attendant 
result that Dissemination efforts and money spent will be wasted 
and that resistance to future Dissemination will develop, 
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3. Climate 

At least two major factors have tended to create a very poor climate 
for Dissemination in the education sector. One factor has been the lack 
of Implementation/Utilization support to the User during the Dissemination 
process. (This we have already discussed above.) Another factor is that 
many products disseminated to school Districts have been of poor quality — 
at least some have been clearly inferior to existing User-developed 
practices, programs, materials, etc. The effect of such Dissemination 
has been to create negative attitudes toward the R5D system and its 
outputs and also a lack of trust in both the information and outputs 
disseminated and the sources of Dissemination information* Under such 
conditions, many Users have become skeptical of any innovation. Further, 
there has developed an attitude that anyone advocating a product is doing 
so for his own benefit or glorification. Finally, the professionalism 
with which the Dissemination was pursued left much to be desired. 

Thus, past Dissemination efforts have created a poor climate for 
Dissemination, a climate which makes it more difficult for new Dissemina- 
tion efforts "CO have significant impact. 

The point to be made in reference to current and future Dissemina- 
tion policy and strategy is that: 

User trust in the source of Dissemination is vital, 
b. To develop and maintain this trust, it is vital to do well 

whatever is done--both in terms of the product and the 

Dissemination efforts. 
Perhaps this is an important part of the reason why commercial products 
are so dominant in school systems. Even though commercial products tend 
not to be the outputs of rigorous, systematic R5D, and not to have been 
carefully tested, the commercial firms do a very professional job of 
marketing and distributing their products. It would seem to be the _ 
responsibility of NIE to ensure that similar care and professionalism is 
built into the Dissemination systems it helps to create. 



167 



4, Numerous but Fragmented Dissemination Efforts 



In the education sector, there has been a considerable amount of 
activity that has been called Dissemination. But still, despite all 
this apparent activity there seems to be widespread disappointment in 
what is perceived to be rather minimal impact on school practices and 
programs. Relatively few RSD outputs produced by the specialized 
Development organizations appear to have been adopted (or i"** adopted, 
maintained) by the User system. Where innovative practices can be 
identified in a school system, they tend more often than not to take 
the form of locally developed exemplary practices or programs that are 
lost to the larger system for lack of packaging and dissemination. 
Therefore, despite all the activity that is categorized under the 
Dissemination rubric, there appears to have been relatively little 
effective Dissemination. Dissemination programs do not appear to have 
been noticeably successful in achieving their objectives. 

There would seem to be several possible explanations for this 
limited impact. But certainly one factor that accounts for the weakness 
of the Dissemination function in education is its fragmented, scattered 
character. Of the several types of Dissemination activity in education 
that could be noted to underscore this point, two in particular illus- 
trate the point well. 

1. Development "Add -0ns" : Particularly in the early years 
of federal funding of the regional laboratories and R5D 
centers. Dissemination was assumed to be an activity 
that should be carried on by each Developer, to make 
potential Users aware of and interested in implementing 
the Developer's own products and programs. In each case, 
a certain amount of the total funding was set aside for 
Dissemination. Such Dissemination "add-ons" tacked onto 
grants or contracts for Development work appear to be 
one of the dominant modes of Dissemination funding in the 
early and late '60s. Such Dissemination was rarely if 

168 

o 

ERIC 



ever carried out by specialized personnel, mechanisms, or 
units with the kinds of training, backgrounds or organiza- 
tional supports that would suggest . strong likelihood 
of success in achieving Dissemination objectives. 
2. Dissemination Channels within Categorical Programs : 
Stronger Dissemination efforts with some greater degree 
of success were (and continue to be) found within the 
domains of the various kinds of federally-funded cate- 
gorical programs (e.g., special education, handicapped 
education, vocational/occupational/career/educations, 
ESEA Titles I and III, etc.)* However, each of the 
categorical programs appears to have developed its own, 
discrete specialized Dissemination system, separate and 
apart from any other progT&m--its own channels, mechan- 
isms, etc. Even if a potential User learns his way 
around the Dissemination system of a given categorical 
program to meet one specific set of needs, he is likely 
to still be totally in the dark about the Dissemination 
channels potentially able to assist him in meeting a 
different set of needs subsumed under a different cate- 
gorical program, 
r-rom the perspective of the potential User, then, there may be 
a multiplicity of potential Dissemination channels that might serve him, 
but he is likely to be only vaguely aware of some, totally unaware of 
others, and too engrossed in his day-to-day operational problems to 
invest substantial time and energies in taking the initiative to cut 
through the morass . 

Federal agencies have become increasingly cognizant of this problem. 
The history of the OE/NIE response over the past decade warrants some 
analysis if policies are to be developed that take into account the 
educational context as it appears today and the various assumptions that 
have shaped that context in the past and may continue (in varying degrees) 
to do do today. 



S, Recent Federal and State Ini t i ative s in Dissemination 

k. Three Modes of Pi s sf ru ination 

Federal Dissemination policy over the past decade or so has 
gone throu?:; three identifiable emphases: first, an emphasis on 
the creation of a c^jntralized, comprehensive resource base through 
which Users could gaiii easy access to needed information; a 
second focus on supporting product advocacy efforts designed to 
• persuade Users to adopt specific outputs of the R§D system; and 
finally, a shift in emphasis to advocacy of change processes rather 
than particular products, with accompanying provision of needed 
supports for the change process. 

1. Creation of a Comprehensive Resource Base 

In the 1960s, federal policymakers came increasingly 
to recognize the scattered, fragmented character of Dis- 
semination in education and the difficulty of locating in- 
formation relevant to a vast array of potential needs, stored 
in innumerable discrete repositories (or not stored at all) . 
The solution to these problems seemed to be to create a 
single, centralized, comprehensive/'generalized (rather than 
specialized) resource base that would meet any User's needs — 
i.e., to place all the scattered resource information in a 
central repository through which, by the push of a button, 
the User would have immediate, automated access to every item 
of information available about his specified need. 

Thus, the massive ERIC system was created by OE to ac- 
quire, store, abstract, and provide easy computerized retriev- 
al of sources from the extensive, unpublished, "fugitive" 
literature of the education sector. ERIC also provided 
publications that announced acquisitions to the field (and 
therefore were expected to make them more visible) , indexed 
the journal literature of the field as well as the fugitive 
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literature stored in the ERIC collection, and provided 
several hundred information analysis products that synthe- 
sized information in selected topical areas. 

Several characteristics of ERIC should be underscored: 

a. ERIC was primarily a repository for Research 
information. 

b. ERIC provided a single, generalized Dis- 
semination capacity. It was not comprised of 

» 

separate, specialized systems targetted at. 
different User groups or focused on different 
problem areas or needs. 

c. ERIC was a passive system. It was a passive in- 
formation repository that required User initiative 
to activate it, and therefore assumed User capacity 
to define their information needs, to leam the 
descriptors, and other tools required to make 
effective use of the ERIC system, and to screen 
and make use of the enormous quantity of (not 
always useful or high quality) output the system 
provided. 

We shall return to the implications of these points later in 
this analysis. 

2. Product Advocacy 

While ERIC was being developed and expanded, a somewhat 
different Dissemination strategy was evolving and being pro- 
moted with federal funding. The annual budgets of the federally- 
funded regional laboratories and R§D centers included specific 
allocations for Dissemination programs, especially those in- 
volving dissemination of the R§D outputs they were producing. 
Various institutions and mechanisms were created specifically 
to advocate the use of particular products or programs they 
selected or developed, and to persuade Users of their merits-- 
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e.g., the Instructional Materials Centers, and Title III 
demonstration centers. 

Over time, however, some difficulties in the product 
advocacy mode of Dissemina'f ion became apparent. Many of 
the products disseminated were of poor quality and were 
over zealously "marketed" by Disseminators functioning as 
advocates. And too, much of the Dissemination effort 
carried out in this mode was amateurish in nature- -it 
lacked polish and professionalism. These two factors 
together — poor quality products and weak Dissemination 
programs--contributed. to an already negative set of User 
attitudes toward the R§D system as a whole. 

3. Change Process Advocacy 

NIE's policies appear to have shifted some of the 
emphasis in federal Dissemination efforts away from the 
product advocacy mode to an advocacy strategy that pro- 
vides supports for change processes and innovation in 
schools as a more general phenomenon. Instead of ad- 
vocating particular products, strategies have been developed 
to: a) provide Users with information about the full array 
of products, programs, information sources, exemplary 
practices, etc. available to meet a given need — comparative 
evaluating information on the alternatives where possible — 
while b) developing the User's capabilities for identi- 
fying needs and evaluating, selecting, adapting and imple- 
menting the products of his choice. The emphasis is two- 
fold: building User system capabilities and increasing the 
"rationality" (in the technical rather than the layman's 
sense) of Users' decisionmaking processes. The State and 
federal Dissemination agencies that carry out this strategy 
provide information and supports that make it easier for 
Users to plan and implement changes (if they want to) . 
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Decisions on whether or not and how to change remain 
with the User. 

B. Some Specific Current Strategy Developments 

Several aspects of current NIE Dissemination strategy warrant 
mention. 

1. Centralized Resource Base: The Expansion of ERIC 

ERIC was designed initially as a resource base of Research 
information. In more recent years, several initiatives have 
expanded ERIC into an informaton base for accessing a wider 
array of resources to meet User needs; ^.§D products and 
descriptive product information packages; exemplary practices 
from the User system; Research and policy interpretations 
and syntheses; guides, catalogs, product inventories, etc.-- 
all input in a form compatible with the automated ERIC 
retrieval system. For those practitioners who make use of 
the system, the kinds of resources that have become accessible 
have broadened in scope, 

2. The States and Intermediate Dissemination Mechanisms 

NIE's Dissemination policies have placed a great deal 
of emphasis on working with and through SEAs and LEAs. The 
federal role is seen as one of facilitating, coordinating, 
and providing seed money to mobilize State and local Dis- 
semination resources and to build needed Dissemination/ 
Utilization capabilities in the SEAs and LEAs. 

Given the scale and variability within the User system 
in education, and the rather limited financial resources of 
NIE, there is simply no way for the Institute to directly 
impact the User system. effectively and significantly. SEAs 
have substantial resource bases to apply to the Dissemination 
function, and in recent years a number of SEAs have exercised 
strong leadership in: developing systems for needs assessment 
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and long-range planning; identifying and disseminating 
exemplary practices; and providing information services and 
technical assistant to local school systems. NIE's strategy 
is designed to support and strengthen the States in these 
efforts — to provide seed money to facilitate and expand what 
they are currently doing, and to facilitate and expand their 
capacity to do what they are doing more effectively so as to 
be able to coordinate education Dissemination activities. 

The pooling of federal and State Dissemination resources 
is expected- to increase the potential payoff of the federal 
investment. The strategy is expected to be more cost/effective 
in the long run, and probably more effective in impact re- 
gardless of costs because of the added possibilities of 
providing services through the States that are tailored more 
adequately to local District needs. 

3. More Active Dissemination/Utilization Stategies 

NIE's State Dissemination Capacity-Building Grants have 
been used in general to establish Cor strengthen) variants 
of a model that includes: a) a centralized resource base 
providing access to the ERIC system, and perhaps other State 
resource bases; and b) a network of- field-based, specially 
trained Dissemination agents Cor "educational extension 
agents," or "information agents") who function as active, 
personalized Dissemination links between the resource base 
and the User. The various States that have created such 
systems differ in how they define the role of the extension 
agent; what services they provide; who they are;, where they 
are placed Ce.g., in regional-type education information 
centers or in the Districts themselves); etc. But regardless 
of which variant is used, the extension agent concept adds 
two elements to the Dissemination process: 

a. There is now an intermediate -level link to facilitate 
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User access to the centralized resource base, 
b. This new intermediate link is an active link. The 

extension agent works with tie User to define problems 
and needs. Information and materials potentially useful 
for meeting these needs are retrieved from the resource 
base, screened, synthesized, and transformed into in- 
formation packages tailored to the User's needs and 
constraints. Follow-up supports and feedback mechanisms 
also tend to be built into the system designs. In some 
of the most active models, the extension agent may even 
function to persuade the User of the need for change, 
and perhaps even provide technical assistance to support 
the planning and implementation of the proposed changes. 
(In addition to the technical assistance capability 
that may be built into some of these State systems, 
some of the newer NIE initiatives such as the R§D 
Utilization Program provide the beginnings of a develop- 
ing technical assistance capability.) 
We might note that all of these more recent strategies result 
from a system-level kind of thinking and that they are es- 
sentially procured to facilitate User access to the information 
resource base. 
4. Two -Way Communication 

Recent federal Dissemination policy statements have tended 
to underscore a view of the Dissemination process as essentially 
two -way rather than one-way communication. The assumption here 
is that if Dissemination efforts are to be effective in ac- 
hieving widescale Implementation and Utilization of Develop- 
ment outputs, more will be needed than simply setting into 
motion one-way flows of outputs from Producers/Suppliers to 
Users. In addition, it is now argued, information from Users 
will have to be fed back into the system— User perceptions of 



their problems and needs, o£ the quality and effectiveness 
of products that are being disseminated, and of the strengths 
and weaknesses of the Dissemination strategies and mechanisms 
being used. Thus, one of the most recent NIE initiatives to 
strengthen the Dissemination function in education is the 
planning of a system currently referred to as the Dissemination/ 
Feed-Forward system. 

C. A Note of Caution 

The newest NIE Dissemination program— the planning of a 
Dissemination/Feed-Forward (D/FF) system—suggests that an 
additional "regional" level of Dissemination mechanisms is about 
to appear on the multi-level scene of local, intermediate. State, 
interstate, and federal agencies and organizations. Although 
still in its early planning stages, it appears that one purpose, 
of the new regional system will be to somehow coordinate "every- 
thing" relevant to Dissemination and Utilization that is taking 
place within a given region. The coordination intent here would 
seem to be to make the various discrete, perhaps scattered 
communication channels and resources in an area more accessible 
to the User. 

There is a potential danger here, one to which we have al- 
ready referred. Certain types of coordination and integration 
may indeed need to occur. But if there is too much "coordination," 
redundancy may be eliminated- -and with redunancy, some of the 
competitive alternatives available to the User will be eliminated. 

Given the likelihood of failure in the education Dissemination 
system (as we noted earlier), the Dissemination system needs to 
be made fail-safe. NIE should not create a single, exclusive , 
intermediate Dissemination channel to link into a given User 
setting. If aU User information in a region must channel through 
a regional agency and the regional agency fails or is ineffective 
with the User, then there is the potential that the whole Dis- 
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semination system may be viewed as tiseless because the User has 
no palatable alternatives. 

If such regional agencies have to be developed (for political 
or other rational reasons of regional needs and opportunities) , then 
this system should be designed in a way that does not eliminate 
diversity, does not result in too much, standardization, does not 
eliminate alternative infoirmation access channels. A possible 
solution might be to take the "regional opportunity" to create 
instead, an alternative system to that pursued by the States, but 
working in coordination with them. 

The orchestration and coordination which is necessary in upper 
levels of the Dissemination system (i.e., in the Agency) needs 
to be subtle. To achieve such subtle yet effective orchestration 
and coordination is difficult because the natural tendency is to 
overcoordinate and overraanage— a tendency that can kill the 
very diversity that is the essence of a fail-safe system aesign. 
NIE will need to be actively involved (and very well informed as 
to the critical issues and impacts) in the system design and 
functioning . 

6. Personnel Base 

We noted earlier how critical it is for Dissemination strategies 
to be carried out in a highly professional manner. In education, however, 
a well developed personnel base of trained Dissemination specialists 
is lacking. Most of those carrying out Dissemination activities appear 
to be practitioners by training. They are proceeding intuitively and 
learning their jobs through hard, often unsuccessful, experience. Several 
federally funded programs have been developed in recent years for training 
D5U (Dissemination and Utilization) specialists. Dissemination mechanisms, 
however, are expanding far more rapidly and creating a far greater demand 
for trained personnel than these programs could ever nope to keep up with. 
This problem suggests policy options in need of consideration— e.g. , 
expanding the training capability and the size of training programs to 
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meet the demand for trained D§U personnel, or slowing the rate of Dis- 
semination system expansion. 

If NIE decides on a policy of facilitating as rapid an expansion 
of the personnel base as possible — to meet the anticipated expansion 
rate of the Dissemination/Utilization function--the Institute may have 
to considei developing recruitment incentives as well as supporting 
expansion of the D§U training capability. In this regard, two points 
can be suggested: 

a. Personnel with classroom and school system backgrounds may 
be particularly well suited for D§U positions. They are 
likely to be sensitive to User needs and constraints, are 
likely to interact well with Users, and, in comparison to 
many others who might be recruited for these positions, 

are more likely to view this work as important and exciting. 
An additional consideration here would seem to be the large 
reservoir of such talent available in the sizeable teacher 
population currently unemployed. 

b. The other likely source for recruitment would seem to be 
persons trained and employed in univv^rsities . However, 
there would seem to be a strong likelihood that such per- 
sons would be oriented more toward an R§D perspective than 
the User viewpoint, would be less sensitive to User needs 
and constraints, less effective in interacting with Users, 
and less likely to find D§U work appealing or exciting. 
Clearly, there are bound to be exceptions, and talented Dis- 
semination agents may be recruited from a wide range of 
settings. But if recruitment efforts are to be adequately 
focused and concentrated, choices will have to be made about 
where the strongest payoff is likely to be. To meet this 
need, we would suggest the User setting as the most appropri- 
ate focus for recruitment efforts. 

7. The Institutional Base 

Within the education sector, one finds an enormous number and 
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diversity of organizations involved in carrying out work relevant to 
Dissemination and Utilization. These organizations will have to be 
taken into account, facilitated, and monitored if the Dissemination 
function in the education system is to be effective. 

The enormity and comple,\ity of this institutional base for Dis- 
semination can be seen by simply developing a list of such organizations. 
For example: 

1. All Producers 

2. Commercial firms 

3. Regional laboratories and R5D centers 

4. Federally funded programs having a Dissemination component 

5. SEAs and ISAs (Intermediate Service Agencies) 

6. Foundations and universities to some degree (perhaps not 
very effectively, and usually in print form) 

7. Consortia of various types 

8. Networks of Users (usually created by the producer of a 
specific product) 

9. Recent NIE funded programs which have created various 
technical assistance mechanisms 

10. Various sorts of organizations that are focusing on R§D 

Utilization (e.g.: Developers/Users; SEAs/Users; User/User; 
etc.) 

These multitudinous and varied organizations must be taken into account 
simply because they are part of -the Dissemination system, even if they 
are not all part of the system created by the federal government. As 
in connection with various other aspects of the R/D§I system, NIE may 
have to accept the initiating role in orchestrating this diverse but 
relevant community, especially if NIE follows (as it should) a strategy 
of building, as feasible, on what already exists in the field. 
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III. IMPLICATIONS FOR AGEN CY ft EMAVIOR 
1. Procurement Behavior 

A. Orchestration as M ain NT H Role 

Given the inraiE : of the educational R/D5I system, the 
large scale and va -ty of the User System, and the frag- 

mented nature of current Dissemination efforts, the key role 
for NIE would seem to be one of system r.^che strati on. These 
very characteristics call for some degree of higher (i.e., 
federal Agency) level orchestration. Beyond this. Dissemina- 
tion is a systems function and can be managed only at an over- 
all systems level. Although OE clearly has some responsibili- 
ties in carrying out the Dissemination function in education, 
NIE is the lead Agency for Research and Development in education 
and is more likely to view R/D§I_ functions in interrelated 
systems terms. Close coordination with OE will undoubtedly be 
needed. But if NIE is to carry out its legislative mandates 
to both "build an effective R§D system" and "improve practice," 
NIE leadership in carrying out the orchestration role will 
clearly be"^necessary. 

This kind of orchestration required by the education sector's 
immature, large and fragmented Dissemination system must be 
carried out in a proactive (as compared to a reactive or passive) 
mode. Orchestration requires an active posture— it will not 
happen if the Agency takes a passive- stance; and it is likely to 
take dysfunctional form if the Agency behaves in a reactive 
manner . 

At the outset. Agency planning must be based on a clear 
understanding of what already exists in the field— what is being 
disseminated; by whom; to whom; how well; and with what degree 
of success as measured by User system Adoption, Implementation, 
and long-term Utilization/Maintenance. Some of this information 
already exists--in scattered sources, reports, and in people's 
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heads (and files) across the multi-level federal. State, inter- 
mediate and local system. Some of it is being gathered in the 
current D/FF system planning effort. However this information is 
gathered- -from contracted studies and analyses, through field in- 
put, as an outcome of the creation and orchestration of a State/ 
regional/national organizational mechanism for Dissemination co- 
ordination--it must then be used by the Agency, taking a somewhat 
directive approach initially, to make decisions (with field 
guidance and advice) as to directions, objectives, etc. for Agency 
orchestration of system facilitation and system building. Given 
the long-term system-building concerns of the Institute, orches- 
tration will have to take on one additional dimension — i.e., 
balancing coordination of current system needs with long-term 
system-building requirements. 

B, System Facilitation and System Building 

Throughout our analysis, we have emphasized system facilita- 
tion and system building as the two overriding objectives of NIE 
Dissemination policy. In the short term, the ^asic need would 
seem to be: a) to facilitate existing Dissemination efforts and 
activities; and b) to fill *'gaps" in the system, eitiier by 
creating new institutions, roles, and mechanisms or by renting 
services. Over the long run, the need is clearly one of system 
building. 

NIE-initiated system design must be congruent with the state 
of the overall Dissemination system (and the educational R/D^^J 
system more broad>). Currently, this means facilitation and 
gap-filling. Over time, this means building into the system the 
capability to adapt, change and even terminate component parts 
as the overall system matures . 

If the system is to function effectively, it must be managed 
in a way that avoids dysfunctional disruptions and permits Dis- 
semination agents to gain the confidence and competency which 
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come from experience and familiarity. Consequently, what is 
needed is both stability and change. To remain congruent with 
overall system maturity over time, NIE-initiated Dissemination 
mechanisms must change. Still, a system must have enough 
stability to permit it to carry out its functions effectiveiy-- 
to build trust, to avoid confusion, etc. Thus, stability too 
must be built into the system design, and must be monitored 
and attended to with great care. 

C. Process Mode of Management Control 

With regard to the Dissemination function (in contrast, for 
instance, to Research) we have suggested xhat NIE must be 
directive in carrying out substantial r ortions of its ov ' i 
role. Whether to control is not the i5;sue. Rathr the 
is what to control. And here, we have focused on the pxu .5S 
mode of managment as most appropriate. 

IVhere th overall system is weak and varied, only the pro- 
cess mode of management would seem to be feasible. Wf.cre the 
overall system is strong, only the process mode is needed. Cur- 
rently, then, the mixture needed would seem be -'ne of : 

a. process control over weak area;; of system functioning; 

b. process control through collaborative coordination 
where strong 'Dissemination agenci s e'xisr. 

D . Linkages/Transforms 

We have noted repeatedly througho it ouv ^^nal/sis V weak c; e 
linkages (or transforms) are between stc\gcs of the R/D>-;i process 
in education. Given the weakness of the kncK ledge and vec" aology," 
base of the field, it is not entirely clear how to transi-or-n 
Research findings into Development products (or even how t^> tran?-, 
form the accumulating Development state of the art. ».nt'5 Develop- 
ment products); how quality control is to be applied to D.: ^^lopmeiit 
outputs; how Development outputs are to be dissemin' .ed to ^sers; 
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how disseminated products are to be used; or even how to 
translate User problems and needs into researchable questions 
or Development product ^specifications or Dissemination/Utiliza- 
tion strategy specifications. 

These weak linkages or transfoTrm? TiUst become a major focus 
of Agency attention if the field's Knowledge/ techno logy base and 
its R/D&I system are to mature. With regard to the Dissemination 
function, this suggests the need for focused effort on: 

a. designing quality control mechanisms for screening 
Research and Development outputs prior to their 
dissemination; 

b. improving the design of Dissemination mechanisms and 
strategies thxough funding. aj:iaiy.sas..ivf..,^ti^^ 

for optimizing User/product/Dissemination strategy 
"fits"; 

c. strengthening the Dissemination/Utilization linkage 
through funding analyses of D§U requirements for given 
product and User types and perhaps too documentation- 
and-analysis Research on naturally occurring variation 
or planned variation e iments in different patterns 
of D5U activity; 

d. design of Researcher/ User and Developer/User linkage 
mechanisms that might strengthen the transforms between 
User needs, problems, preferences, and constraints, on 
the one hand, and, on the other hand, initial Need Iden- 
tification for a given Problem-Focused Research project, 
ongoing Neea Identification (and User/Product "fit") over 
the course of Development projects, and the design and 
implementation of Dissemination/Utilization strategies. 

E. U\^e of Mixed Strategies 

Since the state of knowledge about Dissemination (as a total 
complex process) is low, it seems most appropriate for the Agency 
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to pursue a policy which: 

a. permits a substantial degree of controlled, natural 
variation; and 

b. provides mechanisms for monitoring natural field experi- 
ments and building-in documentation-and-analysis types 
or Research projects to enable us to develop a cumula- 
tive knowledge/technology base from these experiments. 

In addition, given the uncertainty and unreliability of our 
understanding? of D§U processes at this time, and the strong proba 
bility of a significant number of failures, it seems essential to 
provide Users with alternative channels of access to the avail- 
able resource base. Consequently, we have emphasized the need 
for: . 

a. decentralized rather .than monolithic, centralized, 
"over-coordinated" approaches ; and 

b. fail-safe system designs that permit a substantial amount 
of redundancy. 

Furthermore, given the significant amount of variation in 
both products and Dissemination modes in education, there would 
seem to be additional support for an overall Agency orientation 
toward supporting a range of alternatives as well as comb: :/^':v/:r.^ 
(i.e., mixed strategies). 

F. Control Growth and Expansion Rates 

Since the Dissemination system can be built at a faster rate 
tiian User ability to Acopt, Implement, and Utilize what is dis- 
seminated, we have underscored the danger in creating and/or 
expanding Dissemination systems tco quickly. Since such large 
and expensive systems are likely to generate unrealistic ex- 
pectations by Users, Congress, and ME of immediate and widescale 
benefits, the size and expansion rates of NIE-sponsored Dissemina 
tion systems must be controlled carefully. Such systems should 
be no larger, and should be expanded no more rapidly, than is 
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congruent with insuring: 

a, adequate quality control of outputs disseminated; 

b. needed technical support to increase the likelihood of 
User success with disseminated outputs; 

c, a high level of professionalism in all Dissemination 
efforts; and 

d. adequate recruitment and training capabilities to permit 
the trained Personnel base to expand at a rate congruent 
with the rate of system expansion. 

G. Monitoring 

If NIE is to effectively carry out its orchestration role, and 
is to manage the system wherever possible through feedback and 
field guidance, the Agency must put into place a carefully design- 
ed monitoring system, -with monitoring indicators keyed to the 
kinds of pitfalls in system design and implementation we have 
suggested. The monitoring system must be designed well. And 
it must be used in an ongoing, routinized manner to insure con- 
tinual monitoring, feedback, and adaptation--the virtually im- 
perceptible kinds of minor "fine-tuning" adaptations, rather than 
the reactive dysfunctior.ally major system modifications more 
likely to b'^^ the outcome of episodic, discontinous, only occasion- 
al monitoring. 

H. Mandatable Features 

There are several aspects of Dissemination functioni..^ which 
NIE should feel reasonably free to mandate in a sbrnewhat . direc- 
tive manner. For example: 

a. NIE might mandate specific allocations of efforts to 
particular thin market products or to dissemination 

to meet the needs of specific groups not well served by 
existing mechanisms. 

b. It would probably be unwise for NIE to mandate the dis- 
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semination o£ specific products. But it would seem 
perfectly reasonable for NIE to facilitate, or even man- 
date, the creation of conditions (incentives, mechanisms) 
that would make it possible for, or even encourage. 
Producers or Users to become Product Champions for products 
of their choice. 

c. While the decision to buy or rent services may be an ad- 
ministrative matter, conditions that would make "renting" 
a feasible option in certain cases may need to be facili- 
tated. This facilitation may require some Agency orches- 
tration or even mandating. 

d. NIE might mandate certain decisions as to who is to func- 
tion as the Disseminator for a given NIE-sponsored project— 
e.g., whether a given product is to be disseminated by 

its /eloper/Producer or by some specialized Dissemination 
mechanism. 

2. NIE Skills Required 

We have made a number of points throughout this analysis that 
suggest the need for NIF to possess and to exercise great skill in its 
management of the Dissemination function. We have undexscored the con- 
siderable complexity of Dissemination as a total process, regardless of 
the particular sector in which it is carried out. And we have noted 
how the multiplicity of institutional bases for Dissemination in the 
education context, as well as the multiple levels of government involved, 
complicates that picture manifold for NIE. What the Institute needs, 
then, is extensive familiarity with what exists now, and how well it 
works, as well as some rather clear notions about trhe directions it 
might take in system facilitation and system building. 

NIE staff will need to draw on a high level of professionalise on 
matters affecting Dissemi^^ation system functioning. This professions l~ 
ism must come '"ire:: . own internal staff capabilities and the capi> 
bilities it rent^ from various consaltants drawn from academic, R§D, 
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and User (SEA/ISA/LEA) settings. 

Of all the kinds of skills Agency personnel must have to carry out 
NIE's role in the Dissemination system most complex of all are the 
kinds o£ subtle management skills required of personnel who must func- 
tion as "hands-off" planners— subtly, almost imperceptibly coordinating 
and orchestrating at a higher level a system that is highly decentral- 
ized and redundant and that is performing a critical system linkage 
role in an immature R/D&I system in which most of the parts to be linked 
are weak and underdeveloped. We do not mean to suggest that this will 
be anything other than an incredibly difficult, almost impossible task 
requiring skills that probably do not as yet exist— within the Agency 
or outside. 

Still, if our analysis is accepted, this is the Agency role re- 
quired and these would seem to be the kinds of Agency management skills 
to be developed. How such sicills might be developed is another matter, 
outside the scope of this analysis. Clearly, though if NIE is to 
function in the manner we have suggested, the matter would seem to be 
one that merits some attention. 
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I, THE NATURE OF EVALUATION RESEARCH 

1. Defining Evaluation Research 

A> Evaluation Research: Evaluation and Research 

We must note at che beginning of this functional analysis that 
we htx'-^e chosen to use the term "Evaluation Research" instead oJ the 
more common term "Evaluation". We think this is an important dis- 
tinction. The term "Evaluation" leads one to focus on the process 
of analysis and interpretation. While this is indeed probably the 
typical understanding of this function, such an understanding is 
seriously flawed. "Analysis" and "interpretation" require a data 
base that can be obtained only by some form of Research process. 
Thus, this function must be understood as being an interactive 
combination of Evaluation and Research — thus our choice of the term 
"Evaluation Research". 

B. A Unique Function: Knowledge Production and Knowledge 
Utilization Characteristics 

The Evaluation Research function is a somewhat unique function. 
On the one hand, as Research it will tend to have characteristics 
normally associated with the Knowledge Production end of the inno- 
vation process. On the other hand, because Evaluation Research 
is usually carried out for the benefit of Users (administrators 
and policy makers) and because Evaluation Research cannot (by de- 
finition) be performed until after knowledge has been utilized. 
Evaluation Research will tend, to have characteristics normally asso- 
ciated with the Knowledge Utilization trad of the innovation process. 

Figure 4 illust" nes the KP/KU nature of Eva. .at ion Research. 
Figure 4 also indicates that there are different types and purposes 
of Evaluation Research (as we shall note later). Because this 
unique combination of KP/KU characteristics underlies many oil 
the tensions associated with the Evaluation Research functiorij 
it is vitajL that this be understood. 
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2. Two Types of Evaluation Research: Formative and Sumaative 

For many analytical purposes it has been thought useful to sea 
Evaluation Research as having two ..asic forms (though we realize there 
is some overlap). 

A. Formative Evaluation Research 

Evaluation Research-findings may be generated at intervals during 
the life of the program in order to provide a monitoring/self -correcting 
function. As the program will not have been completed, formative 
Evaluation Research findings must be considered to be tentative — 
but they are nonetheless important to program administrators. Their 
function must be seen primarily as part of the program administrator's 
o- developer's team. Sp-n this way, as helpers and not information sources 
for external evaluators, formative Evaluation Researchers need not (should 
not) pose a threat to program personnel. 

B. Summative Evaluation Research 

The basic purpose of summative Evaluation Research is to determine the 
effects of a program through some form of pre/post or time-series or similar 
types of analysis. In this case, findings are generated after the completion 
of a program (or at least, for ongoing or long-term programs, after the pro- 
gram has been in existence for a sufficiently lengthy period of time to 
permit pre/post analysis). 

This pre/post summative Evaluation Research may be performed in order 
to determine how well a program has met its objectives and/or to provide 
a cost/benefit analysis. The change-detecting, summative evaluation may 
also be done simply to identify and understand the effects of a program, 
apart from program objectives and cost/benefit considerations. For 
whatever reason it is carried out, summative Evaluation Research involves 
measurement aimed at detecting changes or impacts attributable to the 
program. 
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C. A Research Dilemma 

It is important now to note that there is an inherent tension in the 
purposes of formative and summative Evaluation Research. 

Formative Evaluation Research seeks to provide evaluative information 
in order that modifications and changes (if needed) may be made in the 
program--actions which break the rules of Research "control". 

Summative Evaluation Research, on the other hand, seeks to determine 
the effects of a program without modifications and changes which would 
introduce "impurities" and reduce Research "control" . 

Persons familiar with Research vTill recognize that the above com- 
parison of formative and summative Evaluation Research is somewhat over- 
stated but nonetheless reflects a very real Research dilemma. It is our 
purpose here simply to take note of this potential tension involved in 
Evaluation Research, not to provide a detailed discussion of this issue, 
an issue that we know has been well recognized in the field in general, 
and by NIE and OE in particular. 

D. The Same or Different Evaluation Researchers ? 

One related issue must also be noted here. In those programs for 
which there is to be both formative and summative Evaluation Research, 
many have questioned whether the same Researcher can properly do both. 
The issue here is two-fold. First, the performance of formative Evaluation 
Research will require some degeee of Researcher contact with program per- 
sonnel at various times during the life of the program. This contact 
can lead the Researcher to develop interpersonal relationships with pro- 
gram personnel and Researcher "interest" in the program-- factors which 
raise the question about the Researcher's capability to provide objtictive 
stramiative Evaluation Research. This problem involves not only Research 
considerations but could be and is often raised for political purposes. 

Second, depending upon the nature of the formative Evaluation Re- 
search design, it may be reasonable to expect the formative Researcher to 
have interpersonal and communication (and perhaps design) skills which are 
not nt .eu by the summative Researcher. 

With these conditions in mind, consideration might be (and sometimes 
is) given to the use of separate formative and summative Researchers. On 
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the other hand, such a strategy would require extra coordination and 
likely be more costly. More typically, in federal social service 
programs however, only suinmative evaluation research has been required, 
and the formative aspects have been ignored. 

3. Three Purposes of Evaluation Research 

To further understand the nature of Evaluation Research and the 
tensions associated with it, we must al*^L^ be aware that Evaluation 
Research has three basic purposes (or uses) purposes which are 
to some degree inherently in conflict. 

A. For Policy Makers and Funders 

One purpose of Evaluation Research is to provide the policy 
makers with information about a program (or project) as a basis for 
decisions concerning future programs. Are the program's results 
(compared to program costs and objectives) of sufficient merit to 
warrant program continuation and/ or expans on? Or should it be dropped? 
Can/should it be modified? If so, in what ways? Can the program be 
used elsewhere? As is, or in modified form? Does the evaluation of 
this program provide insights about other (similar) programs? As can 
be seen, these are post-program questions that are best answered by 
summative Evaluation Research. We note also, however, that in order 
to monitor a program, policy makers and funders may also want infor- 
mation provided by formative Evaluation Research. 

B. For Program Administrators 

A second purpose of Evaluation Research would be to provide 
program administrators with information upon which to base possible in- 
process program modifications and changes. Thus, program administrators 
would tend to be very interested in formative Evaluation Research -- and 
not so concerned with its effects on summative Evaluation Research. 

Indeed, Evaluation Research (both formative and summative) are 
system functions they are a basic part of good ^^ystem operating 
management. The program being designed and implemented should have 
the benefit of what can be learned as the process proceeds. It is also 
vital to have a measure of the impact with the needs of redesign, im- 
provement and future efforts in mind. It is unfortunate that suinmative 
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evaluation Is too often associated with a threat to program personnel, 
so preventing its proper exploitation as an aid to the process. 

C. Knowledge Production 

Simply because it is Research, Evaluation Research can add 
to the knowledge base of the field. However, as this purpose has long- 
term rather than immediate impact, this potential system-building value 
can easily be lost simply by being overlooked by policy makers and fund- 
ers (whose concerns in Evaluation Research tend to be more immediate). 
This value can be lost through inadequate development of the Research 
design, through the selection of less competent Research personnel or 
of "skilled" Evaluation Research personnel who have little interest in 
the substance of the phenomenon being evaluated, and through failure to 
disseminate the findings. It must now be emphasized that the overarching 
purpose of Evaluation Research is to reduce the uncertainty of decision 
makers ~ perhaps most specifically the policy makers /funders who have 
ultimate control and decision power, but also the program administrators. 

4. The Conflict Potential Among Significant Participants 

As the above discussion has implied, there are at least three 
significant participants in the Evaluation Research process: the sponsoring 
policy makers/ the pro&ram administrators; the Evaluation Re- 
searchers. The program staff could be added to '.his list. Because of 
the value-laden political context of education which we discuss later, 
we must also add the public. Each of these participants will have dif- 
fering interests and viex^oints about the Evaluation Research which 
may be in conflict. 

We have already noted the potential for conflict between the 
interest of administrators in formative Evaluation Research and the 
interest of Evaluation Researchers and policy makers /funders in summative 
Evaluation Research. The Evaluation Research literature nptas a further 
potential conflict between administrators and Evaluation Researchers. 
Comments that administrators tend to be concerned with organizational 
stability and survival issues while Evaluation Researchers introduce 
organizational change and growth issues are common. While both concerns 
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are valid organizational goals, the potential for "threat" to the 
administrator exists thereby building in a potential subversion 
of the Research. We can add that the possibility of negauive findings 
may also be seen as a "'threat'' by administrators. 

5. A Real Life Context 

Evaluation Research takes place in a real-life context. It is 
related either to a product which is disseminated into the real 
world or to a program designed to provide people or organizations 
with services. 

Evaluation Research is directly related to decision making pro- 
cesses — either as corrective feedback for administrators or as 
information which can influence policy/ funding decisions. This 
means that Evaluation Research has an "immediate'' time frame in terms 
of usage. 

The management of what is being evaluated (i.e., a product or 
program) is always in the hands of (or at lea^t influenced by) 
someone other than the Researcher. 

As we will note in more detail later, these Knowledge Utilization 
characteristics of Evaluation Research are in (often sharp) contrast 
with the Knowledge Production characteristics of Basic and Problem- 
Focused Research. Knowledge Production types of Research are de- 
signed for the specific purpose of producing knowledge and may be 
considered important in there own right, apart from any potential real- 
world impact. By contrast, the concern of Evaluation Research is 
specifically with the real-world impact both of its findings and 
of the product or program being evaluated; and knowledge pro- 
duction per se must be considered a secondary purpose (though im- 
portant , as we have noted) . 

We will later note that this "real-world context" has some im- 
plications for the skills and experience required of personnel. 

6. The Program Focus of Evaluation Research 

Evaluation Research may be performed in relation either to 
products or to programs. However, most often under government fund- 
ing. Evaluation Research is focused on more amorphous human service 
"programs". Consequently, we shall concentrate the rest of our 
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analysis on that Evaluation Research which is targeted at such social 
or human service programs . 

A. A Specific Focus 

Evaluation Research must focus on a program as the program actually 
is, not as might be suggested by the Researcher or the Research literature. 
Thus, there are significant constraints on the Evaluation Research process 
injterms of Evaluation Research goals, design, time frame, data avail- 
ability, etc. — constraints which Researchers tend both not to under- 
stand and not to accept. This fact can be an impediment for attracting 
competent Researchers. 

B. Program Goals 

Program goals provide the basis upon which a Research design is 
developed and evaluation is made. However, program goals may not be 
simple and clear cut; they may even be the wrong goals. When such 
conditions exist, the Evaluation Researcher would be confronted with 
two significant issues. First, designing the Evaluation Research will 
be difficult at best. Second, there would be the issue of the proper 
role of the Evaluation Researcher with regard to the possibility of 
suggesting changes in program goals. We she. . comment further on this 
issue later. 

C. Program Complexity 

Programs axe rarely simple, especially in human service fields 
such as education. For example, both the content and the outcomes of 
a program may depend as much on the competency and interests of program 
administrators and program staff (or on how well the program has been 
communicated to them) as in the nature of the program itself. This 
fact increases significantly the complexity and difficulty of Evaluation 
Research. 

Similarly, programs will vary. A program may be "broad-aimed" 
or narrow in focus. Where a program is being carried out by different 
groups in different places and contexts, local variation may be per- 
mitted (or even encouraged) as for example in the Follow-Through program. 
Indeed local variati.on may exist even without "official sanction." 
Where local variation does exist, it will be difficult to provide an 
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Evaluation Research design which can validly aggregate dissimilar 

data. Thus, there is ^danger that Evaluation Research will be 

based upon the "lowest common (and probably meaningless) denominator." 

D. The Changing Nature of Programs 

Human service programs are "change-oriented". That is, the 
effectiveness of human service programs is judged by the magnitude 
or significance of changes in human behavior or attitudes ; in social 
settings, in community conditions, etc. The programs themselves 
tend to change because of feedback from participants, from Evaluation 
Research, from environmental forces. Thus the Evaluation Research 
design and methodology requirements may change. Questions of data i^. -o^' 
comparability and aggregation may arisew Additionally, th^rewill' ■ • . 
be the control issue of whether the change being measured results 
from the program or from other organizational or environmental fac- 
tors . 

Research Skills; A Methodological Issue 

For Evaluation Research, different Researcher skills may 
be needed for different kinds of programs. Specifically, where a 
program area is reasonably well understood and the data needs, para- 
meters, appropriate Evaluation Research designs etc. are reasonably 
clear, it would generally be possible for the Evaluation Research to 
be done by Researchers who are competent in standard Research method- 
ology but who have minimal understanding of the program area itself. 

However, where the program area is ox an innovative nature, 
where the program area is not well understood, or where it is not 
clear what are the data needs, the ^Jarameters and the appropriate 
Evaluation Research designs, it would be imperative for the Evalu- 
ation Researcher to be quite familiar with the program area and to 
have 'Skills which would enable him to creace the appropriate Evaluation 
Research design. 

7. Key Characteristics of the Evaluation Research Function 

At the beginning of this functional analysis, we emphasized 
that Research is a basic component of Evaluation Research. Thus, we 
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would expect to find similarities between Evaluation Research and Basic/ ■ ■ 
Problem-Focused Research. However, we also noted that Evaluation^ese'arch 
has characteristics normally associated with the Knowledge Utilization 
and innovation process. Thus we would expect to find disisimilv=?rities 
between Evaluation Research and Basic/Problem-Focused Researcti. Tfo.a program 
focus of Evaluation Research should also lead to differences- iii.,t^^ 
with Basic/Problem --Focused Research . 

Thus, it is important to understand the character is.tics of the 
Evaluation Research function and to understand the similarfities and dif- 
ferences between Evaluation Research and Basic/Problem-Foqused Research. 

A. The Technology Base; Design and Methodologies | 

Evaluation Research and Basic/Problem-Focused Research are in 
many respects quite similar in terms of their design and methodology (the 
technological base). While there are differences, there would seem to 
be more similarities than differences. This is especially true if we 
include within the category of "standard" Research design and methodology 
the process-oriented methods of fields like anthropology and the economist's 
analytical approach which is increasingly used in cost/benefit studies. 
Thus much of the distinction between designs and methods used in Evaluation 
Research and in Basic/Problem-Focused Research appear to involve points 
of emphasis and matters of design implementation. 

B. Purposes 

We have already noted that Evaluation Research has three possible 
purposes: providing information to program poll ::ymakers/f under s; providing 
information to program administrators; and knowledge production (i.e. adding 
to the knowledge base of the field). However, we have also emphasized that 
the overarching purpose must be the providing of information that reduces 
decision uncertainty for decision makers. 'Thus, in. direct contrast to" Basic/ 
Problem- Focused Research, knowledge production per se is a secondary purpose. 

C. Problem Definition 

A basic issue in any Research is control over problem definition — 
i.e., determining what is to be researched. In Basic Research, it is 
clear that the Researcher must define the Research problem. In Problem- 



Focused Research, there needs to., be User/funder input , but the Re- 
searcher is still predominant in translating User/funder problem 
statements into researchable questions. 

In Evaluation Research, the decision maker rather than the 

Eviiluat ion Researcher is predominant ±^ problem definition. .Ii?^<d^§,^4.nM 

we find that funders often define the Evaluation Research problem 
before procuring tha services of an Evaluation Researcher. 

In this context, the issue may arise as to what is a proper 
role for the Evaluation Researcher when he perceives that the problem 
definition of the decision maker is inadequate — i.e., too vague; 
too narrow; would prejudice the outcome; inappropriate for the 
specific program, etc. Must the Evaluation Researcher assume that 
he is an "employee** who merely accepts his "employer's" problem de- 
finition and statement of program objectives, carries Out the task 
specified by his "employer", and then simply provides the "effective^ 
ness assessment" upon demand? 

Or is the Evaluation Researcher to assume the kind of "con- 
sultant/client" relationship with the decision iriakers that would 
allow the Researcher to guide his "client" in developing the problem 
definition and to assess the appropriateness of program rationales, 
objectives and strategies (in addition to measuring program effective 
ness)? The answers to these questions will significantly affect the 
Evaluation Research process from the initial design stage to the 
reported findings. It appears that strong leaders in the Evaluation 
Research field opt for the "consultant/client" relationship, arguing 
that the acceptance of an "employee" type of role by Evaluation Re- 
searchers is a major reason why so much Evaluation Research has been 
of poor quality and has been unable to answer the questions asked. 

D. Clear Targets 

In contrast to Basic and Problem-Focused Research, the "target 
of Evaluation Research is clear. That is, whatever the decision 
maker says is to be evaluated is^ the focus or target for Evaluation 
Research. The Researcher may or may not have been involved in settin 
the target, but there will nonetheless be a specified target. 
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E. Time Frame Constraints 

In Evaluation Research, the time frame is specific (being de- 
fined by the decision maker) and relatively narrow (in order to 
provide timely information to decision makers). This is in contrast 
to Basic/Problem-Focused Research and is likely to be a source of 
tension and frustration to Researchers who have been trained and 
socialized in the university Research environment. 

Additionally, because Evaluation Research is done in an op- 
erational, decision making context, the time lines may shift — 
i.e., decision makers may make "sudden requests" for information. 
Where the environmental context is public and political (as in ed- 
ucation), the probability that time lines may shift is even higher. 

For Researchers trained in Basic and Problem-Focused Research, 
such shifting times lines are likely to be a source of incredible 
tension — both because they would be unused to (and would dislike) 
such sudden demands and because any findings reported under such 
conditions would be incomplete and tentative — and could be in 
error . 

There is a clear implication here for an agency such as NIE. 
The needs of the decision makers and the resistances of Researchers 
both have validity. It would seem necessary, then, that some agency 
ensure that this tension is mediated, probably by building some form 
of preliminary/ interim reports into the Evaluation Research design. 

We must also note an inherent tension in most human service 
program Evaluation Research. Because the Evaluation Research is done 
in an operational context, the needs of decision makers will usually 
require the narrow time we discussed above. However, human service 
programs involve "people change" — a change process that generally 
has a fairly long time frame, especially if community changes are als 
involved. That is, it may not be realistic to measure people change 
adequately or validly within the usually shorter time frames of Evalu 
at ion Research. 
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F> Tighr Management 

In contrast to our analyses of the other functions, Evalu- 
ation Research appears to call for a rather tight monitoring and 
management i.e., it is critical that the Evaluation Research pro- 
cess be timed tc the needs cf decision makers. 

G. Modest Approaches 

The overarching purpose of Evaluation Research is the as- 
sisting of the dec^^^ the level of 
uncertainty in decision making. Further this assistance must be 
provided within a relatively narrow time frame and must be specific 
to a particular program. Thus, in contrast especially to Basic 
Research (but also, to a lesser degree) to Problem*Focused Re- 
search, Evaluation Research calls for a relatively modest approach 

i.e., the objective is to provide "sufficient information" 
rather than "eternal truth". 
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II, EVALUATION RESEARCH IN EDUCATION: A SOCIAL SCIENCE CONTEXT 



Introduction: A Social Science Context 

The history and nature of Evaluation Research In the educational 
context Is virtually Identical with che history and nature of Evalua- 
tion Research In the overall social science context. For that reason, 
we will here focus our analysis on the social science context, refer- 
ring to the educational context per se as relevant. 

1. Historical Development of the Evaluation Research Function 

Of all the R/D&I functions In the education sector. Evaluation Re- 
search has experienced the most rapid and extensive development In the 
last ten to twelve years. 

Prior to the mld-'60's, evaluation of educational programs (when 
It was done at all) was carried out by educational practitioners and 
by some Researchers — but rarely by people who Identified them- 
selves as Evaluation Research specialists. The approaches tended to 
be normative, but rarely systematic or rigorous. The predominant 
strategy was casual observation and analysis. Conclusions tended to 
be based on expert opinion. Intuition, and Impression rather than 
systematically gathered and rigorously analyzed empirical data. 

This pattern changed significantly in the '60s as large-scale 
federally-funded social programs proliferated, and the legislation 
that created them tended to require the systematic gathering, analysis, 
and reporting of empirical data on program effectiveness. Thus, the 
Evaluation Research function expanded rapidly as a new specialty, 
even as a new industry: in less than a decade, the dollar volume 
of federal Evaluation Research contracts expanded at least tenfold, 
with a sizeable portion of the funds being used for Evaluation Re- 
search of educational programs. 

2. A History of Methodological Issues 

During the '60s and early *70s,> there were many heated debates 
among Evaluation and Research theorists about appropriate methodolo- 
gies for the Evaluation Research function. One group argued that ex- 
perimental (or quasi-experimental) designs were more powerful than 
any other Research approaches for assessing the effectiveness of 




programs, products, or strategies — and that it was therefore essen- 
tial to use these approaches to test R&D outputs and social reform pro- 
grams of all kinds. A second group argued that experimental approaches 
imposed unrealistic constraints on field settings and that at any 

rate it could never be possible to meet adequately the statistical, 
design, and treatment assumptions on which experimental approaches are 
premised* 

Other methodological debates revolved around the need for evaluation 
approaches to provide feedback throughout the program development pro- 
cess — not simply telling the Developer at the end of the Development 
process that his program did not work, but working with him throughout 
the process to make it better. Existing pre-post evaluation designs 
made it difficult for program Evaluators to provide this kind of feed- 
back, or to understand how to evaluate a program stimulus that kept 
changing. 

Some of these disagreements have been eased by recognition among 
Evaluation Researchers that there are a number of different kinds of 
evaluation services, each requiring somewhat different approaches and 
techniques. The distinction between formative and summative evaluations 
represents one such distinction. Initially, the same Researchers con- 
ducted both formative and sunanative evaluations, but over time there 
appears to have been some specialization of personnel and organiza- 
tional units here. 

Currently, the formative evaluations that are undertaken as part of 
the R&D program/product development process are generally carried out 
by Evaluators who work with Developers as part of the Development team 
and provide ongoing feedback designed to improve the product or program 
being developed. They use both quantitative data-based and qualitative 
judgmental approaches. Their style of functioning emphasizes flexi- 
bility — changing their Research questions, variables, instruments, 
and approaches as uhe emerging program takes shape and perhaps goes 
through a number of transformations. 

The debate over experimental vs. other kinds of Research designs is 
now centered on summative evaluations -- the evaluations undertaken 
to test the effectiveness of a given program or product after it has 
been fully developed. Summative evaluations are usually done by an 
evaluation agency or organizational unit independent of the program's 
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Developers. Summative evaluations include several types of evaluations 
differing somewhat in emphases because of the different information 
needs of the decision makers to whom they are addressed: 

1. final operational field tests of an R&D output to help the 
R/D&I manager determine whether or not it is ready for 
Dissemination; 

2. evaluations of the effectiveness of a given program or pro- 
duct in a given school or District in meeting locally defined 
objectives ; 

3. evaluations of national program initiatives, sampling prograna 
components nationwide to inform federal policymakers about 
the effectiveness of a given strategy (or the relative effect- 
iveness of alternative strategies) in meeting federally de- 
fined policy goals. 

There is still some disagreement about how appropriate experimen- 
tal designs may be for product tests and for individual school or 
school District program evaluations; and many other kinds of Research 
designs have been proposed for these types of evaluations. Nonethe- 
less, federal program evaluation policy (to whatever extent such a 
policy exists) appears to be moving toward experimental approaches 
increasing numbers of national program evaluations are being conducted 
using experimental designs, control groups, and some randomization of 
treatments. However, the difference between experimental setting in 
the laboratory and the field is gaining recognition. Federal Evalu- 
ators are increasingly acknowledging the need to supplement impact 
data with process data demonstrating that a given "treatment" was in 
fact implemented as specified in the program design, and that the 
impact evaluation is a valid test of the program and not simply a 
"non-event." 

3. The Knowledge/Technology Base 

The early phases of the maturation process of a knowledge and tech- 
nology base are illustrated with particular clarity in the enormous 
literature produced by the Evaluation Research function over the last 
decade. Of all the functional R/D&I specialties. Evaluation Research 
appears to have experienced the most self-conscious and concerted 
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development of its methodology during this period. The literature re- 
flects not only the inherent difficulties of the Evaluation Research role 
and processes, but also the problems of weaning a new specialty away from, 
a parent field. The early literature was filled with self-conscious 
analyses drawing distinctions between Evaluation Research and Basic/ 
Problem -Focused Research and emphasizing the inappropriateness- of pre- 
vailing Research methodology for the educational Evaluation Research 
context. Within only a few years, the distinction from Basic/Problem- 
Focused Research was taken for granted, and the literature documented 
the development of Evaluation Research as a new field with a distinc- 
tive identity. 

The rapid coming of age of the Evaluation Research function could 
be seen in the quick succession of seminal papers produced by Evalua- 
tion Research theorists, the publication of several anthologies re- 
printing important articles on Evaluation Research, the frequent cita- 
tion of the seminal papers of the field and the use of concepts and 
approaches developed in these papers. It could be seen in the emergence 
of a somewhat common frame of ref^rrnce among Evaluation Research theo- 
rists and a common vocabulary -- ,;acluding such terms as "formative" 
and "summative" Evaluation Research and "context," "input," "product," 
and "process" Evaluation Research. The maturation of the Evaluation 
Research function could be seen especially in the formulation of var- 
ious new Evaluation Research designs and methodologies, in attempts to 
develop taxonomies of Evaluation Research designs, and in the publica- 
tion of several handbooks synthesizing and conpressing the accumulating 
knowledge and technology base and translating it into more readily usable 
reference form. 

Still, the conduct of educational Evaluation Research and the qual- 
ity of Evaluation Research outputs have been the focus of considerable 
criticism. The field still lacks an adequate theoretical base. Eval- 
uation Research instrumentation is in a most rudimentary state of de- 
velopment. Basic conceptual and methodological dilemmas remain unre- 
solved. Though substantial progress has been made in recent years, the 
knowledge and technology base of the Evaluation Research function must 
still be considered relatively immature and underdeveloped. 
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4« The Study of Social Change; A Serious Dilemma 

The discussion of methodological issues and of the knowledge/ 
technology base in Evaluation Research provides a context for con- 
sidering a serious dilemma* 

The study of social change is one of the least developed sub- 
ject areas within the social sciences. Our understanding and measures 
of social change tend to be far less incisive and sensitive than they 
need to be, given the fact that Evaluation Research of social change 
can determine the fate of human service programs. 

Thus, the Evaluation Researcher is faced with a serious dilemma. 
He must assess change in a changing context (perhaps changing in 
part because of his presence) — but he must use methods which may 
be inadequately sensitive to the critical changes taking place. In- 
deed, he may not even be ahl^. to 'determine at the time which changes 
are fundamental and critical and which are instead only fleeting and 
tangential i.e., he may be measuring and studying the wrong 

variables . 

While to some extent this problem must be encountered by other 
Researchers in field settings, the problem is rarely as pervasive and 
central for other Researchers, who tend to be less critically con- 
cerned with detecting change and who may have less need to assess 
change in relation to a set of variables so amorphous and changeable 
as the defining characteristics o£ a given human service "program/* 

5. The Personnel/Institutional Base 

The discussion thus far has indicated that there exists today the 
beginnings of a significant Evaluation Research community with its 
own separate identity. This fact is illustrated by the existence of 
journals, books, articles and even university training programs which 
focus specifically on Evaluation Research. 

However, the field is not yet mature and has certain character- 

is tics which must be considered here. 

First, there are as yet relatively few persons well trained specif- 
ically in Evaluation Research, though many more than was the case in 
the mid '6Qs when the sudden upsurge of federal funding for Evaluation 
Research (coupled with much lower levels of funding for Basic and 
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Problem-Focused Research) created a vacuum that wns rather naturally 
filled by Basic and Problem-Focused Researchers — Researchers who were 
trained primarily in psychology and sociology. While these Researchers 
do fill a gap in Evaluation Research, it must .e noted that they would 
tend to be quite frustrated by the relative:.>- formidable constraints 
that exist within the Evaluation Research function (e.g.: time frame 
constraints; the lack of control over problem definition). Such 
Researchers would also have the tendency to redefine Evaluation Research 
problems and questions into Basic /Problem-Focused Research types of 
problems and questions. 

Secondly, the bulk of federal Evaluation Research funding has gone 
to private sector organizations and relatively little has gone to the 
university setting. In the Evaluation Research industry as a whole, 
statistics for HEW-funded Evaluation Research indicate that in 1970 
45% of these federal funds went to for-profit firms; 29% went to not- 
for-profit organizations; 21% went to the universities; 4% went to State 
and federal agencies; and 17% went to individual consultants (Abert, 1971). 

Rossi (1976) has suggested that: naivete in Evaluation Research pro- 
curements has "turned off" the best Researchers. That is, Rossi sug-* 
gests that while it is easy to do Evaluation Research poorly and naively, 
good Researchers have recognized (and thus shtmned) Evaluation Research 
procurements which pose questions that cannot be answered or cannot be 
answered within existing constraints. In a similar vein, evaluation of 
Evaluation Research by the Russell Sage Foundation indicatjas that the 
high quality Evaluation Research tends more often to be done by the academrlc 
community than by the relatively new private sector Evaluation Research firms 
that make up the bulk of the Evaluation Research "industry." 

The existence of an Evaluation Research community (even^;:n6ugh not yet 
mature) .and the evidence that quality Evaluation Research te:? i i to come 
from the academic community would seem to imply the need to reconsider 
Evaluation Research funding policies in terms of selective support to 
facilitate the maturation of a core community of competent Evaluation 
Researchers. 
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6. A Value -Laden Political Context 



Evaluations are often described as management tools designed to 
provide a ''rational basis for decision making" — but decision 
makers in the public sector function in a largely political sphere. 
This fact raises important issues for the Evaluator on both theoret- 
ical and practical levels. 

On the theoretic?^! level, we must ask if political considerations 
are "irrational," or are they based on "a different model of ration- 
ality" from the one generally used by social scientists? 

On the practical level, consideration must be given to the poli- 
tics of decision making. Generally speaking, programs are created 
by political coalitions of diverse interests — interests which 
support programs for diverse redsons. These coalitions tend to view 
negative Evaluation Research findings unfavorably — and generally 
have enough influence to modify or bury negative findings and keep 
their programs going regardless of what Evaluators report. Converse- 
ly (yet similarly), programs may be opposed by other political inter- 
est groups -- interest groups who will use findings of Evaluation 
Research to achieve their ends. Thus, Evaluation Research findings, 
may be used, misused, modified, reinterpreted, buried, etc. — in 
other words, used as a "political football." Given the political 
context and the methodological issues we have noted above, it is not 
surprising that controversies over negative findings often focus on 
methodological rather than substantive issues raised by the findings 
themselves. 

Additionally, we must note that the educational context is value- 
laden, and value choices enter virtually every one of the key deci- 
sions made by the Evaluator. The outcome of Evaluation Research may 
be predetermined by the choice of Research questions and objectives, 
the criteria used in judging effectiveness, and the measurement in- 
struments administered. From the human perspective, the question 
must be asked: Is the Evaluator value- free- when doing Evaluation 
Research? From the organizational/political context perspective, 
the question must be asked: To what extent is/should these key 
value decision choices of the Evaluator be influenced by the organ- 
izational information needs of the decision maker on the one hand 
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and the political context /dynamics on the other hand? 

In contrast, Basic and Problom Focused Research are not sc immersed 
in and vulnerable to a political value- laden context. They do not so 
directly affect the public or decision makers and they tend to be 
remote from the political arena. 

The political value-laden context raises some important issues for 
the Evaluation Research function. Most importantly, it becomes critical 
that potential political issues be dealt with at the beginning of the 
design stage. This would imply a fair degree of interaction between 
the decision makers on the one hand and the Evaluation Researchers. If 
political issues are not dealt with at the beginning, they will most 
likely have to be dealt with later in a highly dysfunctional "attack/ 
defend" context, and significant findings are l/:ely to be muted, ::up- 
pressed or challenged on "methodologxcar' grounds. 

A second ir.portant issue is that unless they are dealt with, poli- 
tical issues are likely to be a strong source of frustration and ten- 
sion for Evaluation Researchers — a fact that can be highly dysfunc- 
tional to the long term health of the Evaluation Research function (and 
thus to the total educational R/D&I system). 

A third potential issue is that Evaluation Research may be used by 
program administrators (and perhaps even funders/policy makers) as a 
"court of last resort," a "panacea" to bail out and save a failing 
program. 

7. Control Over Outcome 

A very strong issue for Evaluation Research that is generally not 
found in Basic/Problem-Focused Research is the issue of who controls 
or "owns" the outcome (the findings). This is both a theoretical/ 
definitional issue, a practical issue and a political issue. 

As a definitional/theoretical issue, we note that because Evaluation 
Research ±3^ Research, the implication would be that the public and the 
educational R/D&I system (through the Evaluation Researcher) should 
have access to the findings of Evaluation Research. Additionally for 
government- funded Evaluation Research, there would seem to be a reason- 
able presumption that the findings are public data. However, we must 
also note that since the purpose of Evaluation Research is to assist 
decision makers, it is they who initiate, procure and receive the find- 
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ings of Evaluation Research, we might assume that the decision maker 
has control over Evaluation Research Outcomes. 

However, there are practical matters to be considered. As the 
one who initiates, procures, funds, receives and uses the Evaluation 
Research findings, it would seem reasonable to assume that decision 
makers have and will exercise control over outcome, especially given 
the political, value-laden context. Thus, for practical purposes we 
must assume that the decision maker controls the findings and that the 
Evaluation Researcher has, in fact, no control over how the decision 
maker will utilize the findings a fact that can be a source of 

frustration and tension to the Evaluation Researcher. 

One additional aspect of this issue warrants consideration. The 
program administrator may have valid reasons for not wanting prelim- 
inary findings to be made public prior to the final summative Evalu- 
ation Research report. Specifically, the program administrator may 
feel that in such a political, value-laden context, the release of 
preliminary findings (lAich may be in error) could lead to unneces- 
sary but seriously dysfunctional interventions in the life of the 
program before the program has had a chance to "mature." On the 
other hand, policy makers and funders may validly request prelimin- 
ary findings in order to monitor the program. 

The resolution of this issue would probably be to have only lim- 
ited circulation of the preliminary findings, with the understanding 
that these would be available for use in the final evaluation report. 

8. The Educational Context 

While the educational context for Evaluation Research is basically 
the general social science context, there are some sectoral-specific 
characteristics that we should note. 

A. Design Variations 

Because of the size and variety within the educational sector (and 
because we do not yet have a fully mature theoretical and methodologi- 
cal base for social science Evaluation Research) , consideration must 
often be given (for the larger-scale programs) to planned or unplanned 
variations in the programs which are being evaluated and thus poten- 
tially in Evaluation Research designs. An example of planned varia- 
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tion is the Follow Through program, which pecmitted local selection of 
model variations. 

Variation in programs and Evaluation Research design may be planned 
or they may be unplanned. Given that the educational system is a 
large, autonomous, decentralized and highly varied system, there is 
really no way to ensure fully that actual implementation will be the 
same as the designed (planned) implementation. 

Whether planned or unplanned, variations in program implementation 
and in Evaluation Research design create significant problems for 
comparative analysis across settings and for aggregation of a data 
base. 

B. Multiple Levels of Government 

Probably more than the other sectors, educational R/D&I involves 
all levels of government. Inherent in this fact is a problematic ten- 
sion for Evaluation Research. On the one hand, decision makers at the 
various levels will have different information needs (including both 
practical and political information needs). To meet these different 
information needs would likely require collection of more and differ- 
ent data than would be needed for decision makers at a single level of 
government. To meet these different information needs would quite pos* 
sibly require a somewhat different (or at least a more extehs ive) Eval- 
uation Research design — perhaps several separate designs. As might 
be illustrated in the case of Title I, there is the danger of reducing 
Evaluation to the trivial level. 

On the other hand, the need of those who fund and make policy de- 
.cisions from Evaluation Research need cumulative, aggregatable data 
which is comparative across settings. There is some trend for govern- 
mental agencies to require certain data to be uniform (e.g.: New York; 
Office of Education) . 

The issue for Evaluation Research designing is thus the extent to 
which varying information needs can be met while still providing cumu- 
lative, aggregatable data which is comparative across settings, 

C. Awareness of Formative/Summative. Evaluation Research Issues 

We have already noted the issues related to fomative and suramative 
Evaluation Research. There was a concern in education that the forma- 
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tive Evaluation Research needs were not being met. The response to 
this concern has been for Evaluation Researchers to work x<rith De- 
velopment teams in educational labs and for some evaluation consul- 
tants to work with school districts in order to provide feedback to 
them. Thus, the education field has developed a certain degree of 
awareness of formative/summative issues. 
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Ill, IMPLICATIONS FOR NIE 

1. Orchestration as the Major NIE Role 

As we have found true for the other functions we have analyzed in 
this study, the major role of NIE in relation to Evaluation Research 
is that of system orchestration. In Evaluation Research, the orches- 
tration will be both complex and vital. The complexity is illustrated 
by the aspects of orchestration which are described below. However, 
probably the major underlying consideration is that given the highly 
political and value-laden nature of the education sector. Evaluation 
Research must be done well and it probably will not be done well 

without NIE leadership in orchestrating the various complexities in- 
volved. In this light, we now focus our discussion on those aspects 
of the Evaluation Research function which require orchestration. We 
now turn to a consideration of these aspects, 

2. Formative and Summative Evaluation Research 

The need for orchestration of formative and summative Evaluation 
Research is two- fold: 

1. There is a tension between the need for "pure" data for sum- 
mative Evaluation Research and the fact that formative Eval- 
uation Research does introduce change into programs. This 
tension will need to be mediated in the designing of the Eval- 
uation Research. 

2. NIE will have to make decisions concerning the issue of 
whether a single Researcher can validly do both formative and 
summative Evaluation Research, or whether separate Evaluations^ ' 
Research organizations are required. If the latter course is 
chosen, then NIE will have to orchestrate the Researchers in 
terms both of the tension between the two modes of Evaluation 
Research and the need to ensure that the data obtained from 

both Evaluation Research organizations is compatible and cumu- 
lative. 

3. Differing Information Needs 

Here, NIE will need to determine for which levels of governmental 
agencies information needs are to be met. To the extent this includes ' 
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several levels, NIE will need to ensure that the Evaluation Research 
findings are cuTOlative, a^gregatable, and comnerable across set- 
tings — and are not reduced to the level of triviality. 

4. The Political Context 

We have noted that Evaluation Research is done in rj, very politi- 
cal, value-laden context, with the result that Evaluation Research 
findings may be controversial and may be distorted, suppressed, re- 
jected, etc. Thus it becomes vital that political issues be dealt 
with in the initial design stage in order to develop and specify the 
way in v;hich negative Evaluation Research findings would be dealt 
with. 

This would seem to imply that there must be communication between 
Evaluation Researchers, program administrators and policy makers/ 
funders in the design process . This would on the one hand tend to 
make the Research Evaluators more sensitive to political realities 
and constraints and would on the other hand tend to make decision 
makers more aware of the danger that Evaluation Research findings 
might .be pre-determined by the Evaluation Research design. Such 
forums of communication would force consideration of Evaluation Re- 
search design objectives and designs and would make the bases of 
decisions more explicit. 

5. Problem Definition 

Final decisions about problem definition in Evaluation Research 
are in the hands of those who procure and fund the Evaluation Re- 
search — not in the hands of the Evaluation Researcher. However, 
the Researcher may be in a position to make valuable (and in some in- 
stances., essential) contributions to the definition of a problem. 
Thus the need is to ensure mediation of a variety of perspectives 
about the problem definition for a specific Evaluation Research 
project. 

6. Are Evaluation Research Findings "Public" Data? 

There is an issue as the whether or not (and to what extent) 
findings are to be considered ^'public" data. This is a political/ 
value decision, and it is not our intent here to debate whether or 
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not government-funded Evaluation Research findings are to be considered 
"public" data. We do want to note that failure to resolve this issue at 
the beginning of the Evaluation Research process is likely to result in a 
heightened degree of political controversy after the Evaluation Research is 
completed. 

One further aspect of this issue warrants further consideration. While 
program administrators and policy makers/f unders need to have preliminary, 
interim reports, widespread dissemination of such preliminary findings could 
result in dysfunctional interventions in the life of a program. This potential 
might be resolved by limiting the initial distribution of preliminary findings, 
but ensuring that they may be utilized in final reports which could have wider 
distribution and availability. 

Interim Reporting 

We have noted both the need of decision makers for preliminary interim 
reports and the potential fru3tration of Researchers about . providing highly 
tentative. findings — especially when the demand comes unexpectedly. Thus, th' 
orchestration role of NIE is to mandate that decision makers have information * 
when they need it; ensure that information is not demanded which it is not 
feasible to provide; ensure that demands are not made unexpectedly on 
Researchers; kad monitor to ensure that the periodically needed information is 
provided the decision maker. 

Additionally, it would seem important that Researchers understand the 
needs of decision makers for preliminary data on the one hand and that de- 
cision makers be aware on the other hand of problems caused when unexpected 
demands are made or when the information requested cannot feasibly be provided. 

Thus, it would seem that a process for providing preliminary interim 
reports should be built into the initial Evaluation Research design. 

Staging 

In the Evaluation Research function, there are several rather clearly 
defined stages which have different basic raquiremeacs . 
A. The Design Stage 

The design stage of Evaluation Research is critical. It is necessary 
at this stage to identify potential political issues and determine how these 
will be dealt with when the Evaluation Research is completed. It is necessary 
to insure that problem definitions and Evaluation Research objectives are clear. 




relevant and within the limits of Research feasibility — and that a variety 
of perspectives are considered in determining the problem definition and the 
Evaluation Research objectives. It is necessary at the design stage to de- 
termine whose information needs will be met and to determine what interim 
reporting is to be provided. 

Because of the critical nAture of the design stage, it is vital that 
the best Evaluation Research talent be obtained — a fact which would suggest 
that NIE must exercise a relatively high degree of control over the selection 
process. 

B. The Data Collection Stage 

Once the problem definition and the Evaluation Research objective? ^re 
clear, there generally would be a whole range of personnel and organizations 
competent to perform data collection. The "best minds" are not necessarily 
needed. Thus, a competitive bidding mode would be relevant. 

There is an exception. In the instance where there is a high level of 
uncertainty about how to define the problem, what methods are really valid, 
etc., the Evaluation Researcher must have a high level of understanding of 
the area to be researched and must be creative and innovative in designing 
the Evaluation Research. In this instance, particular skills are required; 
thus, NIE control over the selection process would be a more relevant strategy 
than open bidding. 

C. The Data Analysis and Reanalysis Stages 

The findings of Evaluation Research are critical both in the sense that 
important policy/funding decisions will be based on the findings and in the 
sense that education is a politically sensitive, value-laden area. It follows, 
then, that it is important to obtain the strong analytic personnel of the field 
for data analysis and reanalysis. In this context, NIE control over the 
selection process would be the relevant strategy. 

For th<» same reasons, data reanalysis is often provided (e«g.: the 
Coleman Report; the Racial Isolation Report ). Thus, a strategy might be con- 
sidered whereby more than one Evaluation Research organization provides 
simultaneous data analysis. In effect, this would be doing data reanalysis 
during the basic data analysis stage. 
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D. Formative and Summatlve Evaluati&a Research 

Formative and summative Evaluation Research require orchestration at 
two points. First, the tension between formative and summative Evaluation 
Research must be mediated. Second, decisions must be made as to whether 
formative and summative Evaluation Research will be done by the same or dif- 
ferent Evaluation Researchers. If done by different Researchers ^ their efforts 
must be coordinated to insure that the data of both will be cumulative, ag- 
gregatable and comparable. 

9. Portfolio Effects on System Capability Building 

Because there are various Evaluation Research organizations already in 
the field, system building per se would not appear to be a primary concern 
here. However, we have noted that while the best Evaluation Research talent 
and the best Evaluation Research work tends to be found in the academic setting, 
federal funding is far more supportive of private sector entrepreneurial organ- 
izations. This would imply a system capability mode of funding which selec- 
tively focuses on the academic sector, and possibly even focuses upon a specific 
set of universities and university-related organizations. 

10. Monitoring and Tight Management 

The emphasis in Evaluation Research is that it must be done well . Thus 
the process must be closely monitored. A "tight management" mode would seem 
to be relevant in order to insure that basic requirements are met — yet care 
must be taken to insure that the management is not unnecessarily and unduly 
restrictive. 

11. NIE Internal Considerations 

A. Relation of NIE to the Field 

The orchestration requirements of the Evaluation Research function 
suggest the need for NIE to have ongoing, close working relationships with 
the field, but do not (in contrast to Basic and Problem-Focused Research) 
appear to require that NIE actually be involved in Evaluation Research. Thus, 
we have suggested, the need appears to be for NIE to have top level personnel 
who have a combination of substantive/methodological skills and political savvy. 

B. NIE Personnel 

As we have noted, NIE must have some top level personnel who have sub- 
stantive and methodological Evaluation Research skills, on the one hand, and 
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political savvy and insight, on the other. Such persons would be able to 
understand the critical implications of: different ways of defining problems; 
different Research methodologies and their relevance to types of programs and 
program situations; and the needs of the different participants — i.e., the 
policymaker s/funders, the program administrators, and the Evaluation Researchers 
Additionally, it is imperative that NIE personnel have skills relevant to the 
process of orchestration. 
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SUMMARY ANALYSIS ACROSS FUNCTIONS AND CONTEXT: 
IMPLICATIONS FOR THE ROLE OF NIE 

INTRODUCTION 

For a federal agency such as NIE, the value of an analysis of R/D&I 
systems is determined by the implications that can be derived for policy 
and strategy decisions — i.e., the identification and evaluation of 
policy/ strategy options and issues. At this point, however, it becomes 
imperative to place some practical limitations upon the nature and 
extent of R/D6el system analyses. 

On the one hand, time and cost considerations make it impossible to 
undertake a ''theoretically complete" analysis the considerations 
involved in a total contextual analysis of all functions and functional 
issues would be horrendously massivfii and complex; the needs of the 
educational R/D6el system are more than NIE could possible respond to; 
there is a vast array of possible options available which are potentially 
relevant to NIE purposes. Thus, the analysis of the system must be 
narrowed down to a feasible scale. 

On the other hand, the analysis must remain sufficiently broad and 
rich so as to provide insight and guidelines for policy and strategy 
decisions. 

Agency personnel need not be overwhelmed by the number and di« 
versity of unfamiliar courses of action suggested. Rather, NIE should 
accept the fact that it cannot do all the kinds of things we have sug- 
gested, at least in the short run. The Agency will be selective (as it 
should be) and determine its priorities. 

To provide d framework within which the Agency can identify and eval- 
uate relevant policy and strategy issues and options, it is essential for 
the Agency to have a basic understanding of: 

1. the R/D&I system as £ system — a system to be understood in 
terms of a total, interactive context which will include the 
system's environment, the operative conditions existing with- 
in the system, and the various system functions; 

2. generic and sector-specific system characteristics; 
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3. similarities and dissimilarities between the various sytr.em 
functions; 

4. the implications of the above for determining appropriate 
Agency policies and strategies. 

The analyses of the four functions will have begun to provide such 
I framework for the Agency. We now turn to a cross-function comparative 
malysis (in summary form) to add to the Agency's analytical/decisional 
framework. 

C. A COMPARATIVE ANALYSIS ACROSS FUNCTIONS; DEVELOPING A SYSTEM CONTEXT 
OVERVIEW 

A cross-functional analysis suggests that the overall requirements 
:ov NIE policies and strategies arise out of three bases: 

1. a sense of NIE's mission as one involving system building and 
various subtle forms of system management; 

2. an understanding of the general immaturity of the educational 
R/D&I System at the present time, what this suggests about the 
appropriate roles to be played by a lead agency with responsi- 
bility for building an "effective R&D system" and what changes 
in Agency behavior might be called for over time as the system 
matures and the capabilities of the field are better developed; 
and 

3. the political, value-laden, social science based nature of ed- 
ucation. 

We shall consider each of these points in turn. 

1. The Mission of NIE 

Before any consideration can be given to specific policies and 
strategies, we must address the more fundamental issue of how NIE 
sees its mission. This issue may also be defined from another per- 
spective. By definition, NIE is a funding agency of the federal 
government. Thus, the key question becomes: What "mission*' per- 
spective will determine how NIE allocates the (limited) federal 
funds under its control? The answer NIE gives to this question will 
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largely determine: 

1. what NIE will and will not do; 

2. what effects NIE will and will not have on the total edu- 
cational system in general and on the educational R/D6cl 
system in particular. 

A. The NIE Mission: Narrow or Broad 

In the most general sense, NIE could define its mission from 
either of two perspectives: 

1. A Narrowly Defined Mission: A Passive Channel for Funding 
NIE could define its mission narrowly as being simply a 
passive channel of funds from the federal government to 
the education sector. Such a narrow definition of mission 
might seem appealing at first glance^ but it provides no 
real basis for making choices of allocation of limited 
resources among multitudinous projects, educational organ- 
izations, etc. At best, this perspective would permit some 
kind of simplified "percentage distribution" formula (e.g.: 
10% to Research; 10% to Development; etc., or 25% to SEA; 
etc.) At worst it ignores the full implication of the neeo 
to have the appropriate quality institutions and personnel 
that should be utilizing these funds. 

2. A Broadly Defined Mission: System Oriented Responsibilitlc:s 
On the other hand, NIE could see its mission more broadly; 
i.e., accepting the responsibility to also use federal Lunds 
to facilitate and develop the educational R/D&I system. This 
system-oriented mission would include concern for system 
building, system maintenance, system monitoring, etc. as well 
as the channelling of funds for direct product procurement 
purposes ~ in a word, a responsibility for determining and 
shaping the nature and direction of the system. From this 
missional perspective, NIE would have the key role of using 
its funding capabilities to orchestrate the various parts of 
the system. 

As the discussion in the previous sections of this report indicate, 
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CISST is assuming that NIE has and accepts the broader, system-oriented 
definition of its mission. Our assumptions are derived from our knowl^ 
edge and study of R/D&I systems in general- our discussions and ex- 
perience with NIE personnel, and our analysis of the R/D&I context of 
the education sector in particular. 
B. Some Implications 

If NIE does not accept this system-oriented mission, then the In- 
stitute is likely to disagree with our systems analyses. More importantly, 
however, the educational R/D&I system is likely to remain innnature, under- 
developed, weak and ineffective ~ with likely long-term negative effects 
for the Institute and the whole educational R/D&I system. Given the nature 
of R/D&I systems in general, and weaknesses of immature R/D&I systems 
(such as education) in particular, some system-wide agency must perform 
a system-oriented role. In the education sector, NIE as the lead agency 
for Research and Development appears to be the most likely candidate and 
probably the only agency with any inclination toward performing this role. 

If NIE does accept a system-oriented mission, then this must be 
made clear — to all Agency personnel, to members of the R/D&I and oper- 
ating system, to Congress, and to the public as well. 
2. Tha State of the R/D&I System 

If NIE accepts a system-oriented mission, then this mission must be 
accomplished within the context and set of needs, opportunities and con- 
straints dictated by the state of development of the educational R/D&I 
system. Effective strategies tend to be context-bound, appropriate to 
the reality of a given set of contextual conditions in a given time and 
place. Thus, policies and strategies cannot be based on abstrations: 
e.g.: that field- initiated R/D&I activity is always "good"; or that 
Agency directiveness is either always "necessary" or always "bad". 

A. Mature and Immature Systems 

Broadly speaking, R/D&I systems may be described as being relatively 
mature or relatively immature. 

For example, in a mature system, one would tend to find: special- 
ization among functions; a basic, solid core of trained and experienced 
personnel within each function; communication networks which facilitate 
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information flow within and between functions (e.g. : journals, "in- 
visible colleges") ; Users who can differentiate between "good" and 
"bad" products; Users who have (or can obtain) the technical capa* 
bilities necessary for successful Implementation/Utilization of in* 
novations; etc., etc. 

In an immature system, one would tend to find the above to be 

lacking i.e., there would be numerous critical "gaps" in the system. 

B. Some Illustrative Implications 

To illustrate the nature and implications of the differences 
between mature and immature systems, let us look at the following 
examples: 

1. System Building 

In an immature system, the weaknesses and gaps of the system 
indicate that system-building needs to be a major policy/strategy 
emphasis. In a mature system, the major aspects of the system already 
exist and are generally functioning well. Thus, in terms of capacity, 
system facilitation rather than system building becomes the relevant 
policy/ strategy emphasis. 

2. The Dissemination Function 

As another example, in an immature system, there is a need 
to "fill gaps" and to develop Dissemination mechanisms. In a mature 
system, one would work with and through existing Dissemination 
mechanisms. Further, we should note that some of the roles and 
mechanisms which would need to be created to serve the needs of an 
immature system would not be needed (and could even become dys* 
functional) as the system matures. Thus, it may be necessary to 
build in change (and even termination) for those Dissemination 
mechanisms which are developed within and for the immature system. 

3. Agency Directiveness Toward the Field 

An immature system will tend to lack self-controlling and self- 
orchestrating (organizing) capabilities. Thus, it is likely that the 
Agency will have to provide a significant amount of direction (though 
how this is done will vary across the functions). In a mature system. 
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less Agency directiveness will tend to be needed because self-controlling 
and self-orchestrating mechanisms will exist within the field (e.g.: 
through the "invisibile colleges''; through quality control mechanisms i 
built into the Development process). 

C. The State of the Educational R/D&I System 

The functional analyses of this study reveal two basic characteristics 
of the educational R/D&I system. 

1. Varied and Large 

The User population in education is quite varied. The User may be 
seen as teachers, administrators, local and State agencies, and (ultimately) 
students. School Districts will vary by size, level of funding, pro- 
fessionalism of teachers, needs (e.g. i urban /rural) ,- etc. . Within a 
single school, there may be varied needs (e.g.: vocational education; 
bi-lingual needs; etc.). The publics of the educational environment are 
also many and varied (e.g.: parents; political groups; etc.). Further, 
the potential Users are numerous — thousands in terms of school districts 
alone. 

2. Immature 

The educational R/D&I system is clearly immature. There are sig- 
nificant gaps in functional specialization. Both the institutional and pers 
nel base tend to be weak. Many of the existing institutions are weak 
or are inappropriately organized for carrying out certain functions. 
Some may have been hurt by previous federal funding policies that shifted 
their nature from organizations carrying out the full range of R/D&I 
functions to more narrowly specialized Development organizations. The 
field has attracted an inadequate supply of first rate talent and much 
of the work that is produced is poor in quality. The field's knowledge/ 
technology base is weak. Communication mechanisms for information flow 
and quality control are underdeveloped. There is little consensus on 
standards for judging the quality of outputs. User system capabilities 
for selecting, adapting, and implementing externally developed innovatior<.J? 
are weak. In all, then, the educational R/D&I system is immature and 
requires Agency behaviors appropriate to an R/D&I system in an immature 
state of development. 
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3> The Nature of Education 

Education as a sector is highly vulnerable to social and political 
influence and is often subjected to considerable pressure from the 
public and from Congress as well as from the varied participants with-* 
in the RScD and operating systems. Schools are public service insti- 
tutions supported by public funds and regulated by public agencies. 
Education by its nature tends to have diffuse goals that are subject 
to value judgments, misinterpretations and controversy goals that 
are harder to specify, less measurable, and harder to use as standards 
against which to judge system performance. Contributing to this vul- 
nerability is the weakness and uncertainty of the field's knowledge/ 
technology base and the public's view of itself as having much knowl- 
edge about education (in contrast to such other fields as health). 

Given the dependence of both the operating system and the R/D&I 
system on public funding, and the generally negative climate that 
has surrounded R/D&I funding in recd-tit years, substantial clarity 
about the Agency's long-term system building role, and its implications 
for Agency procurements and other actions, would seem to be essential. 

In sum, we have suggested that Agency strategies must be developed 
with a view toward system-oriented responsibilities conceived in terms 
of the state of maturity of the educational R/D&I system and the 
vulnerable nature of the educatir^nal sector in general. 

In the following sections we ;rlll be summarizing key issues and 
implications across the &inctions we have analyzed. The issues we 
have already posed will, inevitably, reappear - each in their appropriate 
functional context, giving the appearance of some redundancy. This 
is inevitable and proper given our policy making forms --a focus that 
often finds redundancy preferable to elegance and parsimony. 

II. A COMPARATIVE ANALYSIS ACROSS FUNCTIONS: THREE MAJOR COMMON THEMES 

With the above overview perspective in mind, we now turn to a 
comparative analysis of the four key R/D&I functions which we have 
previously analyzed separately. 

It is important to note that d number of common themes appear 
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across all four functions. At the same time we must also note that the 
operationalization of each of these themes tends to be different with- 
in each function. It is important for NIE personnel to be sensitive to 
these differences. The differences are substantial in terms of some 
of these, less significant in relation to others. We now examine each 
of these common themes in relation to each of the four functions. 

Three of the common themes are of sufficient significance to war- 
rant sepa.-ate attention at the beginning of our cross-functional analysis: 

1. A requirement for NIE leadership; 

2. System building; 

3. Orchestration as the major NIE role. 
1, A Requirement for NIE Leadership 

A. The Need for NIE Leadership 

Given the size, variability and immaturity of the R/ D&I system, 
there is clearly a need for a nationwide agency to exercise a system 
leadership role and to provide direction for the system. NIE is the 
only agency likely to undertake such a role in the education sector. 

B. The Form of NIE Leadership 

The need for NIE leadership must thus be taken as a "given". For 
policy and strategy purposes, the issue now becomes the form of NIE 
leadership. We have noted for each function that the nature of the 
function and the state of the educational R/D&I system indicate that a 
process mode of management will .likely be more effective than a management 
mode based on administration of detailed plans. Even here, however, the 
form of process management will differ somewhat across functions (as we 
shall discuss later). 

C. The Agency vs. Field Issue 

This finding has a specific implication for the Agency/Field 
relationship issue. That is, the relevant issue is not so much one 
of Field- Initiation vs. Ajency-Directiveness as it is one of the way 
in which NIE will provide the leadership which only it .can^rovide. 

D. Functional Areas Differences 

Basic Research 

Given the high level of uncertainty involved in Basic 
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Research, the NIE leadership role will be one of selecting 
Research areas and identifying quality activities in the 
field to support through NIE funding. 

Problem 'Focused Research 

In Problem-Focused Research the key decisions are likely 
to be those involving selection of strong institutions for 
long-term support in an institution building (or re-building) 
mode* 

Development 

In relation to large-scale Development organizations, 
the needed NIE leadership role is likely to be on the early 
stages of Need Identification, i.e., providing some direction 
to these organizations about the kinds of Development out- 
puts needed. In relation to practice-based and practice- 
related Development, NIE leadership will be required to buy 
and/or rent needed capabilities to strengthen this Develop- 
ment mode. 

Dissemination 

As we have noted. Dissemination is a system-creeting 
function and is thus vital for R/D&I system building efforts. 
Since the Dissemination function is relatively weak in ed- 
ucation, NIE will have to provide leadership in the designing 
and building of the educational Dissemination system. 

Evaluation Research 

Two aspects of the nature of Evaluation Research in 
education require that NIE closely monitor and orchestrate 
the Evaluation Research function. 

1. NIE must provide leadership to ensure that potential 
political iasues are resolved (as far as possible) at the 
design stage cf Evaluation Research. Political issues which 
are not dealt with until after the Evaluation Research is 
completed may very likely to destroy any benefits that might 




have been obtained from the evaluation, 

2. NIE must provide leadership to mediate the varying 
information needs of different system participants on the one 
hand (i.e., program managers , funders, and evaluators) , and to 
ensure on the other hand that the Evaluation Research is not 
reduced to a meaningless "lowest common denominator", 

2, System Building 

A, The Need and Implications 

Simply because of the relatively immature state of the educational 
R/D6el system, we have noted in each function that system building is a 
key long-term need. This implies that system building should be one of 
the main criteria for NIE policy/strategy decisions (and thus, also for 
project selection, etc.). This further implies giving consideration to 
multi-purpose and portfolio effects in project and contractor selections. 

The functional analyses also noted that organizations and roles created 
to meet the needs of an immature system may be unnecessary and even dys- 
functional in a mature system. Since organizations tend toward self- 
perpetuation and self-expansion, system-building must include the capacity 
to respond to change — even to termination of some aspects of the system. 

Finally, the functional analyses nota-d that in an immature system, 
the kind of system building that is needed in the long term may not be 
what system members want at the present point in time. 

B, The Field vs. Agency Issue 

The above analyses further illuminate the Field/ Agency issue. Speci- 
fically in relation to system building, NIE will have to assume a major 
responsibility, 

C, Functional Area Differences 
Research 

We have stressed that the system-building process in Research 
is a slow process. Further, the rate at which Research systems 
can be built is limited by the extent of already existing centers 
of excellence. Thus, if more funding is provided than the existing 
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personnel/ institutional base of excellence can productively 
absorb, the result would tend to be mediocre Research and 
the mediocre training of personnel. 

The field has a strong role to play here in working 
with the Agency to guide decisions on which areas should be 
supported, which institutions, etc. At this point in time, 
the system-building purposes of any Research procurement are 
likely to have long-term implications as significant as (and 
often more than) the specific substantive output of the Research 
project itself. The Agency needs to be clear and explicit 
about this, and develop some understanding for this position 
within the Research community, the Congress, and perhaps the 
public more broadly. 

With regard to Problem-Focused Research, a re-building 
mode may be required to restore the Problem -Focused Research 
emphasis to many of the larger scale organizations that have 
been turned into Development organizations through the shifting 
priorities of federal funding in the late '60s. To increase 
the viability of university settings as centers for large- 
scale, high quality long-term Problem-Focused Research, one 
option may be to create joint university-non-university con- 
texts for such Researchers. 

Development 

NIE's system building responsibilities in relation to large, 
specialized Development organizations would seem to be to en- 
sure the development of mechanisms that link these organizations 
to Users. In relation to practice-based/practice-related De- 
velopment work, NIE will have to make decisions about buying 
and/or renting the needed capabilities to identify, package, 
produce and disseminate exemplary programs and practices. 

Dissemination 

Since Dissemination (as a linking function) is system- 
creating, system-building takes on special importance within 
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the Dissemination function. Given the complexities of the Dis- 
semination function and the need to permit the development of 
a redundant full-scale kind of system that provides Users with 
alternative channels of access to the resource base of the field, 
system design Is the key to NIE's role. Given the Immaturity of 
the overall R/D6cl system. Intermediary organizations will be needed 
between R&D Producers/Suppliers and Users but such organizations 
must be designed In a manner that will mandate their '"withering 
away" as the system matures and makes them not only less necessary but 
also somewhat dysfunctional. 

It Is vital to note that the rate at ^Ich the system can 
be established and expanded must be congruent with the much slower 
rate of User absorption of new Information and new Information 
sources. Otherwise, the system will create unreal Is tlcally high 
expectations of rapid and wide scale Impact, thereby leading to 
disappointments that will have long-term negative effects on the 
system. 

All of these requirements suggest the considerable complexity 
Involved in this kind of system building and the need for great 
Agency skill in carrying out this task. 

Evaluation Research 

We have noted that there exists the beginnings of an extensive 
Evaluation Research community with, however, a good deal of var- 
iability among the institutional and personnel bases. We have 
further noted that different stages of Evaluation Research re- 
quire different skills that are likely to be found best developed 
in different organizations. 

Thus, the system building role that is needed to develop 
the Evaluation Research function is not one of system creation 
as much as identifying, developing and orchestrating the existing 
Evaluation Research community. Critical will be the need to select 
those relevant and qualified institutions upon which to build and 
whose efforts should be facilitated. NIE's own quality control and 
selection skills will be critical here. 
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We must also note that the political, value-laden nature 
of educational Evaluation Research underscores the importance 
of having a highly competent Evaluation Research community — 
and thus the importance of system-facilitation within the function. 

3. Orchestration as the Major NIE Role 

A, The Need for System Orchestration 

The functional analyses have indicated from a number of perspec- 
tives, the need for a system-wide and system-oriented agency (i,e,, 
NIE) to provide orchestration for the educational R/D6el system as 
a whole. Such orchestration is needed because of: the system's 
immaturity; its variability; the differences in perspective among 
system members (e.g., Users vs. Researchers); the interdisciplinary 
nature of education; the need to balance strategies in relation to 
the Dissemination function. 

Given the need for orchestration, system monitoring becomes a 
key aspect of the orchestration process. 

B> The Agency vs. Field Issue 

When the primary role of NIE is seen as system orchestration, 
the Agency vs. Field issue changes (or even loses much of) its 
meaning. Given that NIE must provide a system-orientation leadership 
with a key emphasis on system building, and given our understandings 
of the various system functions (as discussed in previous chapters), 
we can see that the real issue is simply who can do what best at 
what given stage in the development of the total system and its 
functions, and bearing in mind the long term system building needs. 
This is often a fine dilemma, one that calls for subtle leadership. 
Thus, orchestration of both its own and field roles becomes the form 
of NIE leadership. 

C. Functional Area Differences 

Basic Research 

In Basic Research, Agency orchestration entails: 
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1. selecting Research areas from among all the possible Re* 
search areas that might be funded (i.e., "placing bets"); 

2. finding and supporting quality activities going on in the 
field (especially the work carried out in centers of ex- 
cellence possessin^^ the necessary minimum critical masses 
of talent). 

Such orchestration can be carried out effectively by an agency only 
if it is staffed by competent Researchers who are actively involved in 
the conduct of Research, and are sensitive to shifts in the field and 
emergence of new Research areas. Basic Research cannot be orchestrated 
and monitored through tight specifications. Rather, it requires the 
presence within the Agency of reseaxchers who are an integral part of 
the field. Advisory panels can be used but cannot be relied on alone. 
Given this close Agency/Field relationship and the active involvement 
of researchers in the field, the Field-Initiated vs. Agency-Directed 
issue tends to melt away. The role of NIE becomes one of s electing 
from a field in which the Agency is closely integrated. 

Problem-Focused Research 

Agency orchestration in Problem-Focused Research involves: 

1. mediating the tensions between the Research and Development 
perspectives, between Researchers on the one hand and 
Funders and Users on the other, and between the perspectives 
of the different disciplines; 

2. orchestrating problem selection in terms of the oft-conflict- 
ing criteria of User needs vs. researchability; 

3, orchestrating the types of institutional settings in which 
the Research is carried out (e.g., large scale settings in 
the private, quasi-public and academic sectors); 

4, monitoring the Research process to make certain that the 
problem focus is maintained. 

As in Basic Research, monitoring of tight output specifications would 
be an inappropriate form of orchestration. 

Development 

Orchestration for the Development function is a very complex 
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matter, both because of complexities within the Developmen.t function 
itself and because Development activities must be orchestrated closely 
with other functions* 

NIE must orchestrate a number of system linkages: Development/ 
Production linkages; linkages between the Developer and the state of 
the art; linkages between a variety of Development institutions; 
linkages to insure User input throughout the Development process, 

NIE must orchestrate the proper roles of NIE and the field in 
Development; i.e., insuring field input in project selection and 
evaluation, orchestrating the selection and placement of advisers, 
emphasizing NIE responsibility in project and system orchestration. 
Project selection in particular requires an NIE orchestration role 
simply because there are so many needs and possible projects — i.e., 
more needs than can possibly be met and thus more potential projects 
than can be funded* Further, the variety of needs indicates a con-* 
sensus building need, which only NIE could orchestrate. 

During the Development process itself, NIE must orchestrate the 
various stages in the process and monitor for quality control through- 
out the process. 

NIE must also orchestrate a balance between the various modes of 
Development; i.e., the specialized Development organizations vs the 
practice-based vs, the technological-opportunity-based modes. 

Finally, NIE itself must know what does and does not exist in 
the field and what NIE itself can and cannot do effectively. 

Dissemination 

Orchestration of the Dissemination function is also very 
complex, but has the distinctive characteristic of having a User focus. 

In the first place, orchestration of Dissemination will be complex 
simply because of the nature of the Dissemination function. Because 
Dissemination is by definition a linkage process, NIE must orchestrate 
the way the Dissemination function informs Users, enables Users to 
utilize innovations, and enables Developers to know what Users need. 
Because Dissemination is a systems-creating phenomenon, orchestration 
of R/D6cl system designing becomes a part of orchestrating the Dis- 
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semination function. Finally, NIE must orchestrate the transforms between 
Dissemination and the other functions. 

Secondly, orchestration of Dissemination will be complex because of 
the variety of possible and desirable Dissemination mechanisms. Not only 
is there no single "right" Dissemination mechanism; a variety of Dis- 
semination mechanisms is needed to provide "fail-safe" for the system 
itself. Dissemination strategies will likely consist of a combination 
of Dissemination mechanisms and methods. There will be a variety of 
intermediary organizations involved in Dissemination. Finally, there 
must be a "fit" between the "what" (products) and the "how" (Dissemina- 
tion mechanisms). 

Similarly, the size and variety of the User market will complicate 
orchestration. Further, orchestration of Dissemination must allow for 
Dissemination to be initiated either by the User or by the Disseminator. 

There are some mandatable features within the Dissemination function 
(for example, allocation of resources to their markets) which must be 
orchestrated with the variety of more naturally energizing Dissemination 
mechanisms. The natural complexity of the function and the need for a 
variety of Dissemination mechanisms and approaches will require an on- 
going NIE monitoring role. 

Evaluation Research 

Orchestration of the Evaluation Research function has three 
key characteristics: it will be very complex; it must be done in a 
value-laden political context; it must be done well. 

Orchestration of the Evaluation Research is complicated first be- 
cause there are many different potential Users and participants. Program 
administrators, Funders, Evaluation Researchers and the public will all 
have different data needs and interests — thereby raising the issue 
and problem of data aggregation. 

Further, there are three possible uses or purposes of Evaluation 
Research: 

1. formative i.e., to provide project management with feed- 

back to help control a project while it is still in process; 
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2. 



suramative -r i.e., data upon which the policymaker can base , 



future decisions. 

3. knowledge production — i.e., adding to the data base of 
the educational sector and the R/D&I system* 

The above considerations raise a further issue for orchestration 
of Evaluation Research i.e., control over outcomes. This is a 
political issue in the sense that various persons may attempt to 
reject, modify or suppress findings; or may seek to use findings 
"out of context" for political purposes. Control over outcomes is 
also both a practical and a theoretical issue. The project manager 
may need feedback during the life of a project, but: 

1. the Researcher may not wish to provide findings which are 
only tentative and may be in error; 

2. feedback data may change the process and therefore make 
impact Evaluation Research difficult at best; 

3. The manager will likeLy not want the feedback data to be 



design stage of Evaluation Research to the extent feasible; 

2. orchestration of control over outcomes to attempt to mlni"^ 
mize feedback made 'piuLIlc during the life of a project 
but that both feedback data and final report data will be- 
come public after the conclusion of the project; 

3. orchestration of the tension between the need of summative 
Evaluation Research or "pure" data and the fact that the 
feedback process of formative Evaluation Research introduces 
process changes which make "pure" impact Evaluation Research 
difficult at best. 

From this discussion, we must conclude that the demands on Eval- 
uation Research are great. Monitoring wili'^De a key pare of "Evaluation 
Research orchestration. 

As if this were not enough. Evaluation Research orchestration 
must also include: 
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made public for fear that it may instigate dysfunctional 
interventions in the project before the project has "matured." 
above discussion leads to three key orchestration needs: 
orchestration to resolve potential political issues at the 




1. the different types of Evaluation Research skills needed for 
standardized as contrasted to highly innovative programs; 

2. the different stages of Evaluation Research which require 
different types of Evaluation Research skills and which thus 
may be done by different organizations; 

3. the issue of having the person who does the operational Eval- 
uation Research (Jand may therefore no longer be objective) 
also 'd(9 the impact Evaluation Research (and If not, as is often 
sOj to orchestrate the two sets of Evaluation Research), 

III, A CCaiPARATIVE ANALYSIS ACROSS FUNCTIONS; SOME OTHER COMMON THEMES 

1, Varied But Close Agency/Field Relationship 

We have tended to stress close Agency/Field relationships through 
out our analysis. Illustrative of the range of relationships we 
see as possible are those that would entail such Agency actions as 
initiating R/D&I activities, or coordinating, mediating, facilitating 
supplementing, or evaluating R/D&I activities already operative in 
the field. 

There is, of course, at least one more option that of man** 
dating where the field will not or cannot do something that is | 
needed. Even here, the Agency would need to have a fairly close 
working relationship with the field to be in a position to know ■ 
what the field will not or cannot do. This Agency/Field relation- 
ship will vary across the functions. 

Research 

In Research, the relationship between Agency and field needs 
to be closest of all functions. It calls for involvement of Agency 
personnel in Research activity so that they can function as an integral 
part of the field, and it suggests the wisdom of a strategy of collabor- 
ative/interactive Research planning by Agency and field together. Under 
such conditions, the Agency vs. Field issue melts away. 

Development 

Development calls for a supportive style of relationship between 
Agency and field, with Agency personnel facilitating quality activity 
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where it exists and building or renting additional Development capacity 
where it does not exist at all or exists at too limited a level. 

Dissemination 

Dissemination is the exception to the general theme of needed 
close relationships between Agency and field. Given the large and 
fragmented nature of the field as defined by this function, there is 
no way the Agency can relate directly to the field. Instead, the 
Agency must relate to the field through directing or facilitating the 
work of intermediaries* 

Evaluation Research 

The political necessity that Evaluation Research be done 
well and the Research aspect of Evaluation Research both indicate 
that NIE personnel must maintain a close working relationship with 
the strong talent in the field. 

2. Multi-Purpose and Portfolio Emphases 

In all of the functions, we have emphasized that each individual 
procurement should be examined not only in terms of its manifest pur- 
poses but also in terms of additional ways the project could or would 
be likely to impact various parts of the R/D&I system, A premium should 
be placed on those projects which not only have important manifest pur- 
poses but which can also provide multiple additional Impacts on the system. 

Similarly, the cost /effectiveness of a given project should be 
evaluated in terms of its overall impact upon and within the total 
portfolios of projects and programs at NIE, across funding agencies, and 
within particular performer organizations. 

Research 

The key to effective Research procurement is the development 
of a bet-placing portfolio. The substantive Research outputs 
being procured must be recognized as having potential long term 
importance, with a very significait aspect of any procurement 
for NIE being its system-building potential. Thus, a Research 
portfolio must be understood as an area portfolio (a portfolio of 
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Research areas In Basic Research, problem areas In Problem-Focused 
Research) • 

Development 

Development portfolios must balance high-risk and low-risk 
projects, short-term and long-term activities, etc. Since the 
substantive outputs of the Development function take on some more 
immediate substantive importance in their own right, this balancing 
of multi-purposes takes on particular significance. Wherever pos- 
sible, potentially synergistic projects should be funded at the 
same time (either by a single agency or in coordination with other 
funding sources). Additionally, multi-purposes should be served 
explicitly wherever possible and balanced adequately in an agency's 
overall Development portfolio. Finally, it is vital for a funding 
agency to be sensitive to the manner in which its funding decisions 
shape the character and capabilities of the Development organizations 
with which they work. 

Dissemination 

In all other functions, we have suggested that portfolio/multi- 
purpose effects be considered along with the manifest purposes of 
a particular procurement. Because of the system-building nature 
of Dissemination, portfolio/multipurpose effects must be considered 
as an inherent part of the manifest purposes of each procurement. 

Evaluation Research 

In Evaluation Research procurements, consideration must be 
given to portfolios which are likely to have long-term system build- 
ing effects. One key question, for instance, would be whether the 
bulk of Evaluation Research funding is flowing to those institutions 
most likely to produce high quality work and develop long-term 
capabilities, or whether instead it is flowing to organizations 
less likely to meet these needs — and what this suggests about 
any weaknesses of Evaluation Research procurements as currently 
structured. 

Additionally, consideration must be given to balancing the 
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varied needs of project managers, funders and Evaluation Researchers 

Finally we must note that Evaluation Research is inherently 
multi-purposeful. That is, it has the three functions of: (a) 
meeting information needs of project managers; (b) meeting in- 
formation needs of funders: and (c) adding to the overall know- 
ledge base of the edr»c£^::on sector. 

3. Staging 

Procurements might be designed in such a way that the carrying 
out of an overall project is broken down into stages (within and be- 
tween functions), providing the Agency with the option to award dif- 
ferent phases of the overall contract (e.g., prototype design/Develop- 
ment/field testing/Dissemination) to different contractors with dif- 
ferent strengths, or to at least provide the possibility for field 
as well as Agency input and review at each step along the way. 

Research 

■ ! ''If'. >^ 

Research: In Basic Research, staging strategies are likely 
to be irrelevant. In Problem-Focused Research staging may be slightly 
more feasible, e.g., by separating into two stages the Need/iden- 
tification/project selection process from the actual conduct of the 
Research.- 

Development 

The issue of quality control provides the basis for staging 
in the Development function. On the one hand, quality control is 
vital to ensure that Users will be able to obtain "good" products and 
thus will develop trust in the R/D&I system. On the other hand, we 
have noted the field self-quality control is weak. Thus, NIE must 
provide for quality control through the creation of mechanisms for 
monitoring of the various stages of the Development process. 

Dissemination 

The system linkage nature of Dissemination and the multiple- 
component nature of effective Dissemination strategies lend themselves 
to a staging process » - 

More importantly for NIE, building the Dissemination system 
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in stages may be both wise and necessary because of the combination of: 
(a) the very large size and the variety of the User target group; (b) 
the cost of building a Dissemination system; (3) the weakness of our 
knowledge about Dissemination as a total process; (4) the need to build 
"fail-safe" into the Dissemination system. 

Evaluation Research 

As noted earlier, the relative variability of the Evaluation 
Research community — make staging necessary. The political context 
makes staging seem a particularly advantageous strategy. Each stage of 
the Evaluation Research process requires different skills. The Research 
design stage is most critical of all and requires bringing in 'the best 
design talent available to design methodologically sound studies able 
to withstand the inevitable attacks on unpalatable findings — often 
attacks that are political in nature but are couched in methodological 
terms. The design stage also seems to he the critical point in the pro- 
cess for identifying, orchestrating and taking into account the diversity 
of value-laden viewpoints that are likely to perceive a stake in the 
definition of "suitable" Research questions and determination of "ap- 
propriate" methodologies — to make certain that what any stakeholder 
might perceive as unpalatable findings is not predetermined by prior 
choices of focus and methodology. 

Whereas in the design stage, the best design talents would seem 
to be required, the picture is rather different in the data collection 
phase, where substantial numbers of competent organizations are__available 
and competitive procurement is likely to be appropriate. In the data 
analysis stage (and perhaps too in a subsequent data reanalysis stage) , 
the best minds would again seem to be called for — - here the best analy- 
tical minds. 

Staging in this manner is likely at this time to produce work of 
far better quality than awarding a single contract for conduct of the 
whole process. Further, there is likely to be a substantial by-product 
in the form of strengthening communication mechanisms in the field (e.g.: 
among the best analytical talent) and enhancing the cumulative development 
of the knowledge/ techno logy base of the field. 



214 



4> Process vs. Product Management Approach 

Given the uncertain character of the field's knowledge/ tech* 
nology base and the professional training and socialization of the 
available personnel, a tight mode of management of R/D&I activities 
is often not feasible. A more reasonable alternative than tight 
monitoring of the R/D&I output or product is monitoring of the R/D&I 
process . This becomes an especially effective mode of management 
if NIE staff function as an integral part of the field while also 
stimulating the field's development of self-controlling mechanisms 
that make an Agency role in quality control less and less relevant 
over time as the system matures. 

Research 

Given the uncertainty inherent in the Research process, 
there is simply no way that the Agency can control the Research out- 
put. Rather, control over the Research process is more appropriate 
and more feasible ~ and is most likely to be carried out well if 
Agency personnel are involved in Research activities, function as an 
integral part of the field, and have close relationships with the 
field. In Problem-Focused Research in particular, where the scale 
of the Research is likely to entail a considerable Agency investment, 
the need for control is high — especially since the capabili^ty^^o^,.^ 
the field is not well developed. However, since this is Research it \ 
cannot be controlled bureaucratically. The process mode of management 
seems essential, but this does require internal Research capabilities 
within NIE and a pattern of relationships with the field that do not 
yet appear to be the general rule within the Institute. 

Development 

In relation to the Development function, the process 
mode of management involves controlling does Development work 
(through project selection) and to some lesser extent how it is done 
through the Agency involvement in some aspects of process design. 

Dissemination 

In Dissemination, the process mode involves control by 
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steering, guiding and overseeing the Dissemination process. 

Evaluation Research 

The process mode of management within the Evaluation 
Research function Involves exercising control at key points In the pro- 
cess. For example: 

1. Intervening during the design stage to Insure that differing 
value-Laden perspectives are taken Into account In the Initial 
definition of the questions to be Investigated and the approaches 
to be used; that evaluation questions are researchable; and 
that Research designs are methodologically sound and likely 

to lead to clear answers to the questions under Investigation; 

2. controlling the types of Evaluation Researchers selected 

to carry out particular Evaluation projects — with the more 
innovative (as opposed to the more conventional) programs 
calling for the kinds of Evaluation Researchers who are cre- 
ative and who also have an In-depth understanding of the 
particular kind of program being evaluated. 

5. Facilitating, Buying, or Renting 

Where needed capabilities exist to some degree within the educa- 
tional R/D&I system, a facilitating/collaborative strategy Is possible 
for NIE. However, where specific needed capabilities do not exist (or 
are too few), NIE Is forced to be more directive. In that case NIE's 
options would seem to be to "buy or rent" the needed capabilities. The 
"buying" strategy would Involve a direct attempt to create Institutions 
with the needed capabilities that would become permanent and largely 
committed parts of the educational R/D&I system. A "renting" strategy 
would entail the temporary purchase of services from organizations that 
already posess the needed skills but are external to the educational 
R/D&I system In that their Involvement tends to be limited and/or of 
short duration. In this case, a "gap" in the' system is "filled", but 
only temporarily, on a project-by-project basis. The advantages of the 
"renting" option are that the needed capabilities are available immediately, 
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NIE can be rather directive about what is to be done, and the funding 
need be only short, specified periods. The disadvantage is that this 
option does not increase the overall long-term capabilities of the 
system. 

Renting may be a preferred option in the case of highly spe- 
cialized or seldom used capabilities that need not be developed internal 
to the system. Additionally, renting may be a useful interim strategy 
— so long as its use does not retard system development because the 
outputs of the emerging internal capability are not as polish>^d and 
professional as those likely to be produced by external organizations 
with well developed capabilities that can be rented. 

Research 

Research : In Research ''buying" strategies tend to be 
most appropriate at this time since the primary consideration in 
Research procurements, we have suggested, is system building. 
While renting is perhaps not a very apt description in this instance, 
cases of attracting Researchers from other fields into educationally 
relevant areas for relatively short periods would be analagous to 
renting. This has been going on in education but should not be a major 
strategy. 

Deve lopment 

In Development, renting or buying are Agency options, 
especially with regard to the generalizing, packaging, production, 
pn4 dissemination of exemplary programs developed in practice-based 
settings. The renting option permits rapid packaging and dissemination. 
The buying option builds long-term internal system capabilities. 
Mixed renting/ buying strategies are also appropriate. 

Dissemination 

In relation to Dissemination the need to build the system 
and the likely high costs of renting suggest the advisability of buying 
strategies. However, the easy replicability of some Dissemination 
modes permits and even favors renting options under certain conditions. 
Mixed renting/ buying strategies may be particularly appropriate at 
this point in the system's development. 
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Evaluation Research 

Here, too, there are options. However the strong system facila* 
tating and building needs suggest the advisability of emphasizing buying 
strategies as the preferred alternative at this time, even though opportun- 
ities of making use of highly qualified and specialized organizations need 
not be abandonned as part of staging strategies. 

6. Mixed Strategies 

Where the state of knowledge about a function is low, it may be 
most appropriate for an agency to pursue mixed strategies that provide 
alternative possibilities and a fair amount of redundancy that is orches- 
trated subtly rather than being "over-coordinated" or "over-managed". 
Often advisable is a policy that permits a substantial degree of natural 
variation along with built-in mechanisms for monitoring natural field 
experiments and using documentation-and-analysis Research to develop a 
stronger knowledge/technology base for the field. 

The need for mixed strategies gets stronger, the closer one 
gets to the User end of the R/D&I continuum, with mixed strategies being 
essentially mandatory in relation to the Dissemination function. 

Research 

The use of mixed strategies is likely to be minimal in 
Basic Research, where "placing bets" on specific Research areas is 
likely to be the primary strategy used. There are more options in 
relation to Problem-Focused Research; e.g. : using small and large 
organizations; strong NIE personnel and advisory panels from the field. 

Development 

Mixed strategies are highly appropriate to the Development 
function; e.g.: mixing use of staging options; single vs. multiple 
institutions; specialized Development organizations vs. practice-based 
Development; practice-based vs. technological-opportunity-type Need 
Identification; production by Developers vs. Users; etc. 

Dissemination 

In Dissemination, the use of mixed strategies is mandatory 
given the variability among products, User needs, User capabilities, 
etc. Mixed strategies are particularly appropriate here since there is 
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clearly no one *'best" mechanism and many mechanisms exist or can be 
created. Use of mixed strategies is the essence of designing fail- 
safe systems. 

Evaluation Research 

The use of mixed strategies here is limited somewhat to 
staging and to selection of different types of Evaluation Researchers 
(e«g«, for evaluation of more innovative vs, more conventional kinds 
of programs). The primary strategies must be: (a) resolving politi- 
cal issues in the design stage; and (b) providing a balance among 
differing information needs. 

IV, A COMPARATIVE ANALYSIS ACROSS FUNCTIONS; SIGNIFICANT DIFFERENCES 
1. Time Frame 

Research 

The time frames for Basic Research projects tend to be 
rather long, while those for Problem-Focused Research may be some- 
what more moderate. Further, since Research involves a high level of 
uncertainty, the Research time frames are generally not predictable. 
System building for both variants of the Research mode tends to take 
considerable time — both to identify and train creative Research 
talent within existing centers of excellence and to build Research 
teams with the needed longevity for productive relationships. The 
rate of expansion for both variants of Research is limited by the 
nximber of such existing centers and their capacity to train additional 
personnel and absorb and use additional funds productively. 

Development 

The time frame for Development projects tends to be short 
to moderate in length and relatively more predictable. 

Dissemination 

Dissemination involves a very different concept of time. 
It requires consideration of the interrelationship between two different 
time lines -- the rapid rate at which Dissemination capacity can be 
expanded and the much slower rate of User absorption and utilization 
of what is disseminated. To avoid unrealistic expectations for system 
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impact, the rate of Dissemination capacity building must be regulated to 
keep it reasonably congruent with expected User absorption rates. An- 
other option of course is to facilitate the work of organizations that 
provide the kinds of User system technical assistance and support that 
might increase somewhat the rate of User absorption and utilization. 

Evaluation Research 

This Function tends to have extremely tight, highly spe- 
cified time lines tied to the information needs of decision makers and 
the time frames of their decision processes. Since evaluation findings 
are gathered for immediate usage as input to these decision processes, 
the time lines are major constraints on the Evaluation Research process 
and cannot be shifted at the initiative of the Researcher. They may 
at times be even shortened at the initiative of the client, as in the 
case when information is needed more quickly than the time frame in- 
itially specified in the Evaluation Research procurement. Consequently, 
the need for interim reports becomes a matter of some importance as 
does the design of Research procedures that permit staged gathering of 
information and formulation of findings. 

There is often a tension in Evaluation Research between the 
immediacy of information needs for decision inputs and the long-term 
nature of what is being evaluated — more often than not the slow change 
process experienced by individuals, social groups, and communities. The 
more socialized the Evaluation Researcher has been in other modes of 
Research (especially Basic Research but Problem-Focused Research as well), 
the more tension is likely to be produced by the time frames demanded 
by the Evaluation Research context. 

2. Excellence 

Research 

In Research, Excellence is the key criterion of judging 
projects and institutions. In Basic Research, Excellence is defined 
in terms of creative, productive, rigorous activity at the outer limits 
of the state of the art. In Problem-Focused Research, the same definition 
applies, tempered to some extent by standards of practicality. 
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Development 

In Development, Excellence is defined in terms of what 
is feasible, practical and usable, not by the outer limits of the 
state of the art. 

Dissemination 

In Dissemination, Excellence is defined in terms of 
effectiveness and professionalism. 

Evaluation Research 

In Evaluation Research, a substantial amount of tension 
surrounds the Excellence issue. On the one hand, it has to be done 
very well (although not in terms of the Evaluation Researcher seeking 
to break the outer limits of the state of the art). However, the 
immediacy of information needs poses serious constraints on what the 
Evaluation Researcher can do. Excellent work that is concluded too 
late to affect decision processes is not useful. Consequently, the 
Evaluation Researcher must produce the best work possible within 
specified time constraints. Interim reports in particular may need 
to be judged with a different standard of excellence from the stand- 
ard used to judge a final report. 

3. Key Criteria for Project Selection 

Ba ltic Research 

For Basic Research in education at this time, the project 
selection process calif's for: 

1. piecing bets on Research areas that are likely to 
bear fruit; accepL:irig the fact that there is likely 
to be no one "right" project; and becoming comfortable 
with a degree of opportunicm in the selection of 
Research areas (particularly being opportunistic 
about the Research strengths of the NIE Research staff); 

?. searching for centers of excellence possessing the 
needed critical masses of Research talent; and 

3. viewing system building considerations as paramount 
in project selection. 
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ProblemwFocused Research 

In Problem-Focused Research, project selection criteria 

must include: 

1. insuring the existence of an adequate problem focus; 

2. insuring the interdisciplinary nature of the perspec- 
tives brought to bear, as needed; 

3. insuring the longevity of team commitment to the problem 
area. 

Longevity is also important in Basic Research, but tends 
to be less of a problem because of the nature of many of the better 
Basic Research personaiel and their personal commitments to areas of 
Research. 

Development 

In selecting Development projects, the Agency must start 
from a posture of accepting the potential equivalence of an array of 
alternative projects. Thus, selection criteria must focus on: 

1. vfaether the projects fall in the priority areas / 

2. whether or not the contractor has the needed capabilities 
and 

3. likely system impacts in terms of system-building, multi- 
purpose, and portfolio considerations. 

Dissemination 

The key project selection criterion in Dissemination are: 
cost/effectiveness in relation to multi-purposes and 
portfolio effects; 

2. providing for fail-safe. 

Evaluation Research 

In Evaluation Research, project selection criteria include: 

1. researchability ; 

2. likelihood of providing information in a form that is 
reasonably immune to methodological attack to serve 
political purposes; and 

3. system-facilitation considerations. 



V. A COMPARATIVE ANALYSIS ACROSS FUNCTIONS; NON -PROCUREMENT STRATEGIES 



AND NIE PERSONNEL 

!• No n- Procurement Strategies 

We have noted that NIE should consider three general types of 
strategies: Procurements, Non-Procurement Behaviors, and Internal NIE 
actions. The* common themes we have been considering are primarily 
procurement-related but can also involve non-procurement behaviors. 
For instance, we have mentioned at several points the possibilities 
for NIE to orchestrate and coordinate funding across the various fund- 
ing agencies — e.g.: federal agencies; funding from other levels of 
government; and other potential private sector sources of funding. 
Multi-purposes and portfolio emphases are as appropriate to the fund- 
ing coordination process as to the examination of individual procure- 
ments or Agency program agendas. Coordination of funding may be used 
to illustrate non-procurement strategies across functions. 

Research 

In Research, cross-agency coordination is vital because 
much educational Research is likely to be interdisciplinary in nature. 
Further, while the proportion of total Research funding provided by 
NIE is small, NIE is the lead agency. 

Development 

In Development, coordination of funding across agencies 
is likely to be helpful but is not so essential. It may be adequate 
simply to know what kinds of Development projects other agencies are 
funding and therefore where the potential for synergy may exist. 

Dissemination 

In relation to Dissemination, coordination across agencies 
may be helpful. However, OE rather than NIE is the lead agency for 
activities that relate to the operating system. Consequently, NIE 
may have to work very closely with OE in these coordination activities. 

Evaluation Research 

Given the extensive amount of Evaluation Research activity 



that is supported by federal agencies and other levels of government, 
coordination of such funding would seem to be helpful both for orches- 
tra tiorfpurposes and also for developing a cumulative knowledge/ tech- 
nology base for the Evaluation Research function. 

2, NIE Personnel 

We have noted that to orchestrate, monitor, and relate to the 
field in each function, NIE needs to have personnel with specific types 
of skills and backgrounds. 

Research 

If NIE is to relate to the field in the manner we have 
suggested, it will be essential for NIE to have on its staff Researchers 
who are from the field and vrtio continue to be involved actively in the 
Research process. This would seem to be an essential requirement for 
effective NIE process monitoring and orchestration of the field, and 
even more critical if Research planning is to be carried out in an 
interactive/collaborative mode. We have suggested that the kinds of 
Researchers needed here are not the "stars" of the field who are likely 
to distort the field to their own image of where the field should be. 
Rather Researchers are needed who can facilitate and work with the field, 
who are sensitive to developments going on throughout the field. These 
requirements would seem to contribute significantly to the case for NIE 
conducting a limited in-house Research program not in competition with 
but as a contributor to the field. 

Development 

The kinds of NIE personnel needed to manage the Institute's 
Development programs would seem to be education professionals sensitive 
to the needs and constraints of the education context and ^o also possess 
the kinds of skills needed for effective orchestration and facilitation. 

Dissemination 

If the Dissemination function is to be carried out in a 
highly professional manner, it would seem advisable to have at least some 
Dissemination professionals on the staff of the Institute talent that 
can likely be "rented" temporarily or acquired on a more permanent "buying" 
basis. 



Evaluation Research 

The Internal Agency skills needed here would seem to be 
political as well as methodological insight and savvy regarding such 
matters as different ways of defining problems; the different kinds 
of Research methodologies that are relevant to different kinds of 
projects; the varying needs of program managers, f under s, and Evalu- 
ation Researchers; and the relevant potential political implications. 
Most likely this mix of skills is to be found only in experienced, 
first-rate Evaluation Research talent. 

In concluding this section we feel that it is appropriate 
to echo a theme first raised in the Preface. NIE priorities and re- 
sponsibilities can not be only a reflection of the Agency's budget 
profile. As a small agency in a large field its activities as co- 
ordinator, orchestrator and facilitator, involving non-procurement 
based efforts are likely to have very significant implications for 
the short and long term health and functioning of the educational 
R/D&I system. 

3. Structures 

A cross-sectoral comparative analysis reveals some differences 
across functions in the forms of NIE organizational structures that would 
be most relevant to each function. 

Basic Research 

The uncertainty involved in working at the outer limits of th 
state of the art is the key to determining relevant NIE structures in re- 
lation to Basic Research. NIE cannot orchestrate, guide and monitor this 
field from centralized structures and detailed plans. Rather, NIE needs 
to have personnel who know the field: what is being done, where and by 
whom, in order to determine which areas of concern to concentrate upon and 
which personnel and institutions to support in a long-term system-building 
mode. These conditions suggest decentralized, emergent-organic structures 
with generally open boundaries between NIE and the field such chat NIE 
personnel are actually a part of the field. This would seem to call for a 
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relatively loose distinction between NIE/field organizational membership foi 
NIE personnel — for example, as might be achieved through personnel inter- 
changes and leaves to and from universities (as is now occasionally done) . 

Problem-Focused Research 

In comparison to Basic Research, there is relatively less un- 
certainty in Problem-Focused Research, though this is still a significant 
factor. The boundaries between NIE and the field are clearer, but there 
remains the need for NIE personnel to be directly involved in Problem- 
Focused Research, Since both NIE and field personnel would be involved 
in Problem-Focused Research (though probably separately) , the key element 
of structure becomes some form of linking mechanism between NIE and the 
field. A matrix structure might be relevant in teirms of the interdisci- 
plinary nature of educational Research and the various problem areas of 
Problem- Focused Research — and to facilitate selection of institutions 
to support in a system-building mode. 

Development 

Development is not an in-house NIE function. NIE's role is 
to provide direction through funding and monitoring. Since Development is 
a product-oriented function, NIE might consider structures which are organ- 
ized according to product typologies. 

Dissemination 

Because of the size and variety that characterizes education 
and because we generally cannot say there is "one best way" for all situa- 
tions (or even for a particular situation) , diversity and even redundancy 
characterize Dissemination. Thus, there will be structural diffusion — 
i.e., several different structures may emerge in the field and be supported. 
NIE structures must be supportive of, and congruent with, field structures. 
One relevant NIE structure could be a problem/geographical matrix organiza- 
tion. 

Evaluation Research 

The key needs for Evaluation Research are (a) orchestration of 
complex set of needs, relevant participants and perspectives; and (b) insur- 
ing that Evaluation Research is done well. This situation presents a key 
structural dilemma. 
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On the one hand, the politically sensitive nature of Evaluation 
Research requires that it be done well and that potential political issues 
be dealt with in the Evaluation Research process. Further, the various and 
possibly conflicting needs and perspectives of the participants require 
orchestration. Finally, especially with regard to sunnnative Evaluation 
Research, it is the policymaker/f under who must and will determine problem 
definitions and Evaluation Research objectives. These conditions would 
seem to call for close control at a high level within NIE. This would also 
seem to call for top level NIE personnel to have both substantive and 
methodological skills and political savvy. 

On the other hand, the Evaluation Researchers need to have a 
good deal of freedom in matters of Research methodology. Further, there 
need to be ways for the Evaluation Researchers to provide guidance in mat- 
ters of problem definition and Evaluation Research objectives. Additionally, 
program administrators need to be able to obtain the information they need. 
These conditions would seem to call for significantly less control by NIE. 

Perhaps it is best to say that NIE must insure that basic needs 
are met by the Evaluation Researcher (e.g.: that some level of basic 
Evaluation Research standards are me\y that certain NIE-specif ied objectives 
are accomplished; and that the politi^^^^l issues are dealt with construc- 
tively — but that beyond these basic requirements, there should be a good 
deal of freedom and flexibility within the Evaluation Research function. 

VI. CONCLUDING REMAJUCS; COST CONSIDERATIONS IN RELATION TO POTENTIAL 
LONG-TERM INVESTMENT PAYOFF 

We would be remiss in concluding this analysis without mentioning 
the costs likely to be incurred by the kinds of strategies we have proposed. 
In comparison to the procedures currently used, it is likely to be con- 
siderably more internally costly for Research programs to be developed 
through long-term, intensive, collaborative/ interactive relationships be- 
tween Agency staff members and the field. To take another example, stag- 
ing of' procurements (especially when this involves complex orchestration 
of diverse viewpoints in some stages) is likely to appear to be a more 
costly alternative than awarding a contract for the whole project to a 
single institution. 
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We acknowledge this cost factor. But we also point out that the 
considerable amount of NIE staff time currently invested in the research- 
ing and writing of RFPs was one of the concerns that prompted the Council 
and the Institute to request this policy analysis. The added cost may 
also be more apparent than real when this is offset by potential gains in 
productivity. 

We have no way of putting a price tag on the kinds of options we 
have proposed. But if NIE accepts the mission and the role we have sug- 
gested, the costs will have to be absorbed and referenced in terms of both 
increased effectiveness and efficiency and the potential long-term invest- 
ment payoff in building future system capacity. The point need not be 
labored further. 

The case for NIE to assume system-oriented responsibilities is, 
we believe, a strong one, entirely consistent with its legislative mandate 
and its position as lead agency for Research and Development in education. 
We suggest that the rather fundamental and broad-ranging questions raised 
by the Council and the Institute, once subjected to analysis from an R/D&I 
systems perspective, demand at the very least thoughtful consideration of 
the case for a 4:ather substantial restructuring of the Agency's relation- 
ships with the field. 
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SCENARIO ANALYSES 



I, NIE-DIRECTED COMPETITIVE PROCUREMENTS; A SCHOOL DISTRICT SURVEY 

1. Description of the Procurement 

2. Impacts 

3. Evaluation of the Procurement and its Impacts, Actual and Potential 

4. Possible Alternatives 

5. Summary Description of Alternative Procurement and Related Activities 

6. Likely Impacts 



II, UNSOLICITED PROPOSALS; A PROPOSED FRAMEWORK FOR DESIGN OF NATIONAL 
PROGRAM EVALUATIONS '■ ' " 

1. Description of the Procurement 

2. Impacts 

3. Evaluation of the Procurement and its Impacts, Actual and Potential 

4. Possible Alternatives^ 

5. Summary Description of f^temative Procurement and Related Activities 

6. Likely Impacts 
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SCENARIO ANALYSES 

INTRODUCTION 

Two illustrative scenario analyses follow: 

1. an examination of the implications of alternatives to 
NIE-Directed Competitive Procurements, where less 
competitive modes might seem more appropriate; and 

2. an examination of the unsolicited proposal as a pro-* 
curement mechanism and options that might be consid- 
ered in funding R/D&I activities initiated through 
the unsolicited route. 

Consideration of these two issues in particular was requested by NIE 
staff. 

The specific scenarios we have developed both concern procure- 
ments of Research. In part, this is because both issues seemed to 
lend themselves to richest illustration in the Research context. But 
also, it seemed to us that Research thinking is central to NIE pro- 
curements as they are made currently. If we could be persuasive in 
making a case for system-oriented thinking in relation to the Research 
function (where much of what we have been suggesting might seem particu- 
larly alien) , then it might be even easier at a later date to make a 
case for this pattern of thinking in relation to other functions. 

Each scenario is presented in accord with the analytical model 
described earlier in our introductory chapter (see pp. 19-21 and es- 
pecially Figure 3) : 

1. a description of a procurement (in both cases these 
are hypothetical procurements but based to some degree 

on typical cases ) ; 

2. analysis of the likely impact of a procur^x>ent im- 
plemented that way, given the educational R/D&I context 
at this point in time; 

3. evaluation of the possible strengths and weaknesses of 
the prociarement based on the above analysis; 

4. developiiie.nt of alternatives to (or modifications of) 
the initial pattern of procurement that might better 
achieve Agency purposes; 
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5. description of the modified or alternative procurement 
approach, its latent and manifest purposes, and its 
implications for non-procurement activities and actions 
within NIE; and 

6. analysis of the likely impacts of this alternative pat- 
tern of procurement. 

It should be noted that both scenario analyses presented here were 
hypothetical rather than actual cases. Clearly, the scenarios would be more 
useful if they were based on actual cases of NIE procurements — past, present 
or contemplated. We would have preferred to develop a series of scenarios 
based on actual cases. However, this would require some discussions with NIE 
project officers and other NIE personnel, as well as analysis of relevant 
documents and perhaps too, discussions with others outside of NIE involved in 
the plannin- and/or implementation of the particular procurement. We would 
hope to conduct such empirically-based scenario analyses some time in the 
future if NIE judged this to be desirable. 
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I. NIE-DIRECTED COMPETITIVE PROCUREMENTS; A SCHOOL DISTRICT SURVEY 



1. Description of the Procurement * 

In 1975, NIE issued an RFP for a survey of 2000 school Districts. The 
objective was to be able to identify those school Districts that may have had 
some involvement in locally generated innovation. The survey was to gather 
basic descriptive data on these school Districts, who they were and what kind 
of things they had done, size, age, location, personnel, student population, 
SES characteristics, etc. ~ so as to develop a better understanding of the 
conditions which affected the innovation behavior of those school Districts 
that would be so identified in regard to both internally and externally devel- 
oped innovations. One of the central questions of concern was whether these 
particular school Districts, having already demonstrated some capacity for 
self help might not represent a highly receptive and possibly qualified pop- 
ulation for R&D products that had been developed externally. If so then, 
potentially, these Districts might well merit becoming the target of more in- 
tensive dissemination and technical service efforts to supplement their own. 
In turn they might become the source of model programs, of exemplary practices. 

On the other hand, it was also possible that, given their own, self 
generated efforts they would be resistant to innovations deriving from ex- 
ternal sources (the Not-Invented-Here syndrome). Further, it would be im- 
portant to know whether such conditions held generally (or not) across the 
total population of such school Districts or whether they might vary with 
respect to such factors as: 

size 

urban/rural 

minority concentration 



*While the case is presented to read as though it describes an actual NIE pro- 
curement it is in fact hypothetical. Also the authors imply no position on the 
substantive content of the procurement. 
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professionalism of staff 
District structure etc. 
Critical dependent variables to be observed as indicators of 
potential receptivity might be: 

scope of adoption behavior (rates, scale, etc.) 
maintenance after initial trials 

observable levels of institutionalization (e.g., after 
periods beyond 3-4 years) 

The study was seen as the first step in what might become a long 
term survey program aimed at monitoring these effects so as to permit 
a responsive strategy on the part of the various delivery and dif- 
fusion mechanisms that might be operating in relation to these Districts. 
Thus, this initial survey was viewed as a critical first step that 
would develop the sampling frames that would permit improved later 
studies and also both natural and planned field experiments (e.g., 
to examine alternative dissemination and/or technical assistance 
strategies), as well as to permit experimentation with varying modes 
of Development (e.g., specialized Development Organizations vs. 
Practice-Based) . 

The core of the issue was how local innovation activity inter- 
acted with external based R&D. A particular sub-concern involved the 
special case of "Basic Skills". How might the conditions implied above 
affect the likely response to products being developed in this area 
in the RfitD system and, relatedly, what could be done to make such 
products or programs more adoptable and useable by this type of 
potentially very important target (in terms of the planning of a 
major development and delivery program)? ' 

The survey was to be conducted in two phases 

Phase 1 - Survey of the 2000 Districts selected as a carefully 
statified sample of the total population. The objective of 
this phase was to identify which of the Districts had exhibited 
what kind of internal innovation behaviors, and their behavior 
with respect to externally based R&D products. 
Phase 2 - Was to be a more intensive survey of the sub-group 
identified in the phase 1 survey as having had a history of some 



significant internal innovation activity. The objective of this 
phase was to increase our knowledge of the nature behavior and 
attitudes of such Districts. The size of the second sample would 
be determined based on the phase 1 findings — but with the hope 
of having a population of some hundreds. 

Based on the two phases it was hoped that the characteristics of 
school Districts likely to have been involved in different kinds of 
innovation, with varying intensities, could be identified, as well as 
specifically identifying a sample known to have been engaged in such 
practices. Further, some specific indications might then be obtained 
as to the responsiveness of this sub-group (and hopefully therefore of 
the larger group of school Districts that had demonstrated self-help 
capacities). 

The primary purpose of this procurement was substantive — - i.e., 
to gather the desired data. But there were other purposes involved as 
well: system-oriented purposes (in terms of developing baseline data 
for system monitoring and developing a listing of innovative school 
Districts) and environmental purposes (e.g., demonstrating that 

that had been expressed by associations of Chief State School Officers 
and other LEA personnel \9ho would make up a substantial proportion of 
the overall survey population). 

A point to be noted about this procurement is the extent to which 
it partakes of characteristics of both Problem-Focused and Evaluation 
Research (or Policy Research). It was intended to increase our under- 
standing of a potentially important group of Users/Innovators in the 
educational R/D&I system (about whom we know relatively little at this 
time), to permit NIE to carry out its system management role more ef- 
fectively. And too, it was oriented toward meeting information needs of 
policymakers suggesting an opportunity area in the system in need of NIE 
policy initiatives, and providing the beginnings of a time-series data 
base against which to evaluate the impact of the Institute's policies 
and programs. Consequently, such a procurement would seem to call for 
a balancing of the requirements of both Problem-Focused and Evaluation 
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Research — a uension not easily mediated between a) the creativity 
needed to develop a useful picture of a little understood system and 
b) Agency specif icatioD cf the key infonnitcioii needs to be met. 

In the za^e of this particul?;: procurement, the Agency issued a 
highly specified RPP — specifying the variables to be studied, the 
questj.ons to be an;iwc>red, etc. and requiring frequent and extensive, 
highly specified intCirim reporting in accord with a specified schedule. 
A considerable amount of NIE staff time was invested in the preparation 
of the RFP (as well as in the prior wcrk to develop the stratified 
sample of the 2,000 school Disr.ricts to be surveyed in phase 1). 
There was some interaction with SEA/LEA sector representatives (whose 
cooperation was essential to achieving a high survey response rate), 
but little if any interaction between NIE and other highly knowledgeable 
participants in the educational R6cD system — this despite the fact 
that NIE had some contracts with a number of organizations studying 
aspects of the educational R&D system and specifically conducting 

surveys of school Districts — perhaps out of concern that such interaction 
with these contractors with whom the Institute already had a relationship 
would be construed as giving them unfair advantage, — a violation of the 
"fairness" principle so strongly clung to in government procurements, where 
all potential contractors in a system are expected to be given absolutely 
equal treatment in their relationships to the Agency. We shall return to 
this to this point shortly. 

The RFP appeared to be a call for a single, well qualified 
survey organization to carry out a predesigned survey and provide 
prespecified data analyses. One month was allowed for the response. 

2> Impacts 

a) The Agency received a substantial number of proposals in 
response to the RFP, mostly from single contractors (rather than 
collaborative contractor/subcontractor arrangements), mostly proposing 
to carry out the work as specified rather than suggesting alternative 
types of surveys, mostly from large-scale private sector survey research 
corporations in short the competitive mode was reasonably successful 
in attracting a good number of the types of organizations considered 
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most appropriate to carry out this procurement. 

b) Costs: NIE invested considerable staff time in the stages pre- 
liminary to issuing the RFP and in the evaluation of proposals to select 
the survey contractor* Perhaps 20-30 organizations expended resources 
(equal perhaps to 1 to 2 man-months each) in responding to the RFP and 
in subsequent contract-related interaction with NIE. 

c) The survey contract was awarded to a single highly qualified 
organization xd.th strong survey capabilities and substantive knowledge 
of issues relevant to understanding the innovative behavior of schools 
and school Districts. Agency purposes, then, would seem to have been 
achieved well. 

3. Evaluation of the Procurement and its Impacts, Actual and Potential 

In evaluating the procurement, Agency personnel must weigh actual 
(or likely) impacts of a procurement as initially formulated with possible 
desired impacts. The question to consider here is: How might such a 
procurement be designed to better achieve Agency purposes, or to achieve 
a broader range of Agency purposes? 

At the very least this question requires insuring some clarity on 
Agency purposes: What in fact are the Agency purposes tTiat are viewed as 
relevant to this procurement? Might a somewhat different view of the 
Agency role and its requirements for procurement policies produce a 
different or an enlarged picture of the relevant purposes? 

If (S) NIE's role is conceived largely as being a funding channel, 
with J^gency leverage on the system exercised primarily through choices 
as to which work it funds, and if (b) the work to be procured is relatively 
straight-forward data-gathering in accord with the prespecified design — 
then the procurement mechanism (the RFP) would seem entirely appropriate. 
However, if (a) NIE conceived its role primarily in terms of long-term 
system building, and if (b) the Agency started from the premise that 
the extent and processes of innovation in school systems are not well 
understood phenomena and that developing a data base for understanding 
such processes required the input of creative thinking from the R&D 
field and not simply more mechanistic kinds of conventional data-gathering 
(which is more appropriate to well understood phenomena) — then other 
Agency procurement strategies become reasonable alternatives to consider. 
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If the survey is viewed as primarily data-gathering, i.e., to 
identify 

a) a specific group of ''self help'' innovator schools 

b) the structural and other characteristics of such schools 

c) limited and minimal data on prior adoption behavior in the 
respect to external innovations 

then competitive RFP-type procurement may indeed be the wisest course 
of action. Strong Research talent might be "turned off" by the RFP's 
specificity, but this might be quite appropriate since the Agency might 
view it as undesirable to divert strong Research organizations into con- 
ventional data-gathering of this type. Given the substantial number of 
competent survey research organizations attuned to RFP procurements who 
could effectively gather the desired data. Agency purposes would indeed 
be well served by use of a procurement mechanism likely to attract these 
survey organizations and not the strong researchers whom the Agency 
would rather see doing other kinds of work for which they are more 
uniquely qualified. 

However, if the survey is thought of in terms of designing a data 
base and monitoring system that will be useful in the long run for de- 
veloping an understanding of a weakly known and little understood 
group of school Districts, and developing the kind of understanding 
that will be useful for identifying policy-relevant leverage points 
as well as achieving a degree of system building, then it would seem 
that a case could be made for a different pattern of procurement that: 

a) could attract strong Research talent to the design stage of 
the survey effort; 

b) could have some impact on strengthening such strong Research 
institur:ions and shaping their long- terra agendas and port- 
folios ; 

c) developing powerful communication linkages among the Research 
talent that might be considered an "invisible college" on 
school innovation processes and possible synergy or collaboration 
among their separate efforts, etc.; and 

d) developing a close positive working relationship between NIE 
and this part of the field, with NIE's internal personnel 
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coining to be viewed as an integral and key part of the field, 
facilitating quality work in the field and development of the 
field, 

(If the procurement is seen in somewhat creative terms, then NIE 
should be fully cognizant of an impact that seems entirely likely if the 
work is procured as it was in a tightly specified RFP. Such procurement 
forces creative Researchers with procurement sawy and understanding of 
the complexity of the task they are undertaking into a form of game- 
playing with the Agency: contractors suggest that they will generally 
abide by the RFP's specifications — for they will not otherwise win the 
contract — while in their own minds they are fairly certain that the com- 
plexities of the task make such scheduling, specificity of outputs, etc, 
infeasible and likely to be modified substantially in the course of conduct- 
ing the survey) . 

If NIE procurements in general came to be viewed in multi-purpose 
terms, and in relation to possible, supportive non-procurement activities 
and internal NIE actions, then the procurement might be carried out in 
conjunction with: 

a) NIE initiatives to coordinate its survey with similar work 
elsewhere (e.g., in OE) , to seek cross-agency collaboration, 
synergy, pooling, of resources, exchanges of information, etc., 
and 

b) internal NIE actions to promote a systems perspective among 
personnel in program units and to develop a pattern of cross- 
Drogram communi elation and synergy that might strengthen Institute 
functioning as well as total NIE impact on the R/D&I system 
toward which its somewhat discrete programs are directed. 

Thus for example, in this particular area a number of NIE groups 
would (or should) have substantial interest. These would include those 
concerned with .Dissemination and Feed Forward processes, with R&D utili- 
zation, with local problem solving, with the overall R&D system and with 
Basic Studies, even though only one of these groups was actually involved 
in the procurement being discussed, and was in fact pursuing the project 
independently of the others. 
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4, Possible Alternatives 

A, Dev eloping a close working relationship with the stronn^ talent in 
the field during the survey design sta^e; 

Use of the RFP mechanism for procuring this work probably had the 
effect of cutting NIE off from the base of creative Research talent 
who tend to be "turned off" by RFPs — who could not conceive of carry- 
ing out Research they did not themselves design (or at least play a 
^ole in designing) or in working within tight specifications as to 
deadlines, outputs, etc. Many of these Researchers may not even be 
aware of the existence of this Research program (so that they might 
think about participating) since they are unlikely to read the Com- 
merce Business Dally or other such sources where competitions generally 
more attractive to entrepreneurial organizations are announced, 

L£ the procurement is thought of (initially, or after the evaluation 
stage described above) as requiring input from creative Research 
talent — to zero in on what kinds of things we need to study :>o as 
to achieve long-term policy-relevant goals (for example that night 
involve the exploitation of diffusion processes based on seeding school 
districts having high implementation success probabilities, based on 
their demonstrated self help capacities)* then perhaps the survey 
design stage could be viewed as a wedge for achieving a wider range 
of Agency purposes. By bringing the strong talent in the field to- 
gether in an ongoing relationship with the Agency, NIE might use the 
design of this survey as the basis for long-term program planning of 
the Research program to be served by (and to further the developsient 
of) the monitoring system. This kind of relationship with the field 
might affect not only the long-term NIE Research program agenda/port- 
folio, but also influence the shaping of long-term agendas of these 
strong Research organizations. The greater the involvement of these 
organisations in the design of this long-term data system, the greater 
the likelihood that synergy will develop within and between their own 
organizations' Research agendas. Thus, this mode of relationship nas 
the potential of being the wedge whereby NIE may shape the portfolios 



*We wish to remind the reader of the hypothetical character of both 
the process described and our neutral position as to the suDst<3ntive 
content. 
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of the institutions; that appear to have the strongest potential for 
producing important quality work in this area. The capability-building 
potential here may in fact have greater long-term consequences than the 
particular survey (or even the particular monitoring system) being de- 
signed. 

.If a survey of this kind is to be an element in a continuing 
process, some kind of ongoing close relationship with the field of po- 
tential contractors might be essential. Otherwise, each contractor 
carrying out a piece of the long-term program is likely to pursue its 
own conception of what is involved here (which may at times be appropriate, 
but within bounds of some sort), or NIE will be forced to be highly directive 
each and every time about the work to be carried out. 

B. Staging the procurement with competitive RFP-type procurement used 
only in the data-gathering stage : 

The procurement might be staged so that (1) R£cD specialists 
work collaboratively with NIE during the design stage (either as "con- 
sultants" or as the result of modified competitions such as invitations 
to them to submit proposals of design ideas, etc.); (2) the larger 
scale phase 1 data-gathering stage is handled competitively through 
RFP-type procurement; and (3) the more intensive but smaller scale 
phase 2 survey and the data analysis and subsequent reanalysis stages 
might again be handled in only modified competitive form (e.g., inviting 
specific innovation process specialists to submit proposals) or as open 
competitions that include an element of actively encouraging certain 
organizations to submit proposals. With regard to the analysis stage, 
what we are suggesting may again come into conflict with the "fairness" 
principle. If some form of phase 2 and/ or data analysis competition 
pits specialists involved in the design stage and/ or the phase 1 stage 
against the rest of the field, it is highly likely that someone will 
cry "foul" and argue that'any of those involved in the initial efforts 
should be barred from the later analysis competition because their know- 
ledge gives them "unfair" advantage. The other possibility — closing 
this competition to only those involved in the design stage (or perhaps 
specifically inviting them to bid) might produce even louder cries of 
"foul" since clearly they are being treated differently from the rest of 
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the universe of potential competitors » The problem is not easily 
resolved. 

C« Issuing a competitive procurement-announcement that uses the NIE- 
developed survey design as illustrative, and encourages creative 
design work and collaborative arrangements between organizations 
strong in survey research and others strong in understanding 
innovation processes 

Another option rather different from what we have been suggesting 
would be to implement the procurement as a single procurement but 
change the specificity, time frame, etc. to make it possible for more 
creative design work to be carried out in the proposal development and 
to facilitate the development of collaborative arrangements among 
organizations with talents suited to different phases of the overall 
procurement. 

It should be noted that while collaborative contractor/sub- 
contractor arrangements were not prevented in the initial RFP, nor 
was a contractor prevented from suggesting an alternative design for 
the survey (in addition to bidding on the design specified), such 
possibilities were made somewhat difficult by the one month or less 
available to the contractor to draft the proposal. Collaborative 
arrangements take time to develop and complicate the problems of 
writing a coherent proposals Furthermore, developing alternative con- 
ceptions for the survey -- different in conception, design, execution, 
etc. from what was suggested in the RFP — is likely to be difficult 
if not impossible within the RFP response time frame, especially given 
the varied other demands on the time of those drafting the responses. 
In all likelihood, considerably more time was invested by NIE personnel 
in the development of the design specified in the RFP. Clearly, any 
organization responding to a competitive procurement is aware that 
investment of time and money in responding to an RFP is a gamble with 
payoff likely only in a fraction of all cases. Therefore, the time an 
organization is willing to invest in responding to an RFP is likely to 
be limited — not sufficient to conceptualize, elaborate, and present 
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^ell a design for an alternative approach significantly different from 
that specified in the RFP, 

There are problems inherent in this option, then, and perhaps a 
staging here might resolve this — e.g., competitively procuring brief 
statements of creative design work proposed, then selecting from these 
a small number of organizations to work with internal NIE staff to develop 
the designs within a reasonable time frame. 
Coordination within NIE 

Given the fact that the survey data to be gathered have potential 
utility across NIE program boundaries, and that some of the data needs 
of other NIE programs might have been met by this survey, then planning 
of this survey might have been used as a wedge to promote cross- divisional 
and cross-program communication within the Institute— to promote a more 
systems-oriented pattern of thinking across the Agency as a whole, 
especially to seek synergy and interaction effects across program lines. 
Clearly, the Institute would be well served by such a strategy. 
^» Summary Description of Alternate Procurement and Related Ac t ivities 
A. Staging/Field Agency Relations 

The procurement could be usefully thought of as having tne following 
possible elements: 

1) the initial conceptualization and general research design 

2) the first phase mass (2,000 school districts) survey 

3) data analysis and the selection, of the sample of "self-help*' 
school district innovations 

4) the second phase intensive survey 

5) the data analysis for the second phase and possible reanalysis 
of the first phase data 

6) design of alternative delivery strategies 

7) design of potential field experiments. 

While these elements would not all need to be so separated, it 
is readily evident that different skills ^re required as between various 
combinations of elements. Also, there are likely to' be critical questions 
that will need answering at each stage before it will obvious as to 
whether it is justifiable to proceed. Thus it will be only after element 
3) above (the phase 1 data analysis) that we are likely _o know whether 
it is indeed feasible to identify self-help innovation districts through 
such a survey; whether there are any reliable predictor characteristics 
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that can differentiate schools not only as to the general question 
but as to the types of innovation in which they engage; whether a 
sufficient nximber of such districts can be identified so as to make 
phase 2 viable; and most important, whether the phase 1 findings 
give any indication of the "self-help" factor as likely to account 
for a variance in the reception to external R&D products. Similar 
milestone requirements would exist after elements 1) (initial design) 
and 5) (phase 2 analysis). 

These two aspects, the variable skills required, and the 
critical milestone requirements lead us to consider staging as a 
potentially desirable strategy, as follows: 

Stage 1 . Initial conceptualization and research design could be 
carried out as part of a loosely defined grants compe- 
tition. The NIE staff would take responsibility for 
synthesizing the outcomes in collaboration with a panel 
of consultants drawn from the several Researchers (or groups) 
that were chosen from the competition. 

Stage 2 . An RFP, relatively tightly drawn and based on the initial 
synthesized conceptualization, would be ^^ed to select and 
fund a single contractor skilled for the phase 1 survey 
and preliminary data analysis. 

Stage 3 . After completion of the phase 1 survey and analysis the 

NIE staff, in conjunction again with the panel from Stage 
1 above, would make the determination as to whether to 
proceed with the study. 

Stage 4 , If the decision was positive a second RFP would be issued 

for phase 2 and the continti.ing analysis. This RFP would be 
far less tightly drawn— only mandating the use of the phase 
1 results in the planning and design. These results would 
be a part of the RFP as would be the original synthesized 
conceptualization. The RFP should be open to everyone 
(possible) and collaborative relations between groups strong 
in conceptualization and in survey research could be 
encouraged by indicating in the RFP the importance of 
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Stage 5 . 



creative conceptualization of the issues and by permitting 
the maximum possible period for response (and informally 
by NIE letting the field know that this is seen as a 
desirable aspect). 

Using personnel identified in the previous stages as 



active consultants, NIE sould take responsibility for the 
development of alternative delivery strategies. RFPs 
could later be issuer for the design and conduct of field 
experiments and evaluation research (formative and then 
summative) when this becomes appropriate. 



B. Multi-Purpose Strategy 

As we have already noted the opportunity for achieving several 
NIE purposes through this procurement exists. 

The manifest purpose of identifying the "self-help'^ Districts 
and of learning something of their innovation receptivity behavior 
has been described. But there are other more latent purposes that 
could be achieved. 

1. System Building 

At least two opportunities for system building can be identified. 

First, the very targets of the survey, the self-help Districts, 
could be looked upon as potentially very important members of the 
educational R/D&I system (as sites for model programs, possibly 
receptive entry points for R6cD products, models of local implementation 
capacity development, etc.). It might therefore be worth considering 
the development of a loose network of such Districts— operating much 
as an invisible college might. Attempts to involve them in programs, . 
even in later stages of the present study, could have positive consequence 
for the R/D6cl system. 

Second, we could consider the consequences for the R/D&I system 
of developing the network of Researchers who might be involved in 
this study. The effect of creating the original design group and 
their continued use as a panel of consultants could be very important 
for the field. Also, the efforts aimed at linking the more conceptual 
Researchers with the strong field survey organizations for work in 
this area (as suggested above) could act to add considerable capacity 
to the educational R/D&I system. 




2. Monitoring 

The issue of monitoring as a critical requirement for R/D&I 
system building and orchescr^;r.ijn was frequently noted in our general 
analysis. We have also already touched upon this in our review of 
this case. While monitoring was not an initial or explicit purpose 
of the study, it is evident th-^ ^ make use of its findings (if 
these turned out to be of val the design of future dissemina- 

tion and development strategi j.d demand an ongoing monitoring 

process. Additionally, the type of data that wou** d be collected would 
prcvrVIt: basis for the design of a system to moniuor the existence 
anc . -'oning of a potentially very significant phenomenon in the 
R/DSci ..... em. Hence, in orchestrating the survey design, execution 
of its phases, etc., NIE should make explicit its desire to have the 
efforts lend themselves to these monitoring purposes. 

3. Environmental Impact 

It may be that by paying explicit attention to the value of 
self-help efforts, and presenting in a positive light NIE's desire 
to sea a synergistic interweaving of internally and externally- 
ganerated innovations » that positive consequences could be generated 
for the Agency and its R£cD efforts. These might involve increase 
in legitimacy ("We do not assume that all wisdom comes to you from the 
outside") and in the political climate ("We are really seeking to involve 
quality local capacity"). 

C. Internal NIE and Other Non-Procurement Behaviors 

The central theme of the proposed NIE strategy with respect 
to this study was that frequently recommended in this analysis-- 
orchestration. The key to the strategy was the careful orchestration 
of various types of participants so that they could play their optimal 
roles at the right stages, the building-in of key checkpoints or 
milestones, the encouragement of desired collaborations, etc. 
Clearly this would call for NIE personnel capab'' a of performing such 
tasks, as well as the synthesis of ideas for the study as we recommend- 
ed after stage 1. 

We .also noted the need for 'cooperation and communication across 
a number of groups within NIE that could (or should) have been inter- 
ested in the study. Further, NIE might consider with which other 



agencies (e.g., OE) it should be cooperating in the conduct of this 
study and In what mi;;;ht grow out of it. 

Finally, there may be other (essentially) non-procurement actions 
that would be seen as contributing to the various purposes of the 
study. One such example might be the holding of a session at AERA on 
the topic, or a special conference. There may be others. 

6. Likely Impacts 

We have suggested throughout this scenario what we believe might be 
a number of the likely iispacts of this alternatively designed procurement. 
We summarize these here: 

a) There would seen to be a strong possibility of highly creative 
thinking in the survey design stage. 

b) Consequently, the long-term data base and monitoring system 
evolving out of this survey program would likely be approcsriately 
oriented toward: developii j j.ncreased understanding of the R/D&I 
system in education; and identifying useful leverage points for 
policy interventions. 

c) Several system-building impacts seem likely: developing commu- 
nication linkages around the strong Research talent of the rele- 
vant Research areas (e.g., "invisible college" mechanisms); 
providing additional support for, and facilitating the quality 
work of strong Research organizations working :'n relevant Research 
areas; affecting the long-t:c?nii Research agendas of these strong 
organizations and increasing the^pctential for sjmergy across the 
work of these various organizations (and between their agendas and 
NIE's); and possibly over the long run developing linkages among 
the self-help Districts and strengthening their capabilities as key 
participants in the R/D&I system. 

d) NIE's close working relationship to the field in carrying out this 
procurement and related activities is likely to strengthen NIE's 
position vis a vis the field and enhance the Agency's image as an 
integral and key part of the field facilitating its development. 

e) Othv- positive environmental impacts would also seem likely. SEA and 
l£A personnel are likely to view with favor NIE initiatives that 
recognize their innovative potential and accord them respected status 
as key organizations in the R/D&I system. Given the generally strong 
influence of education interest groups on Congress and the public, this 
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would seem to suggest important side-benefits for the Institute's 
image, legitimacy, and long-term stability. 

f) NIE's position as the lead agency for Research and Development in 
education is likely to be strengthened by cross-agency coordination 
of the kind suggested here. Given the critical interrelationships 
between OE and NIE programs, it would seem clear that collaborative/ 
coordinating strategies between OE and NIE (as well as other agencies) 
should have substantial long-Lerm payoff. 

g) For NIE, one of the most significant organizat: onal impacts of this 
alternatively designed procurement may be the promotion of internal 
communication across NIE divisions, the development of synergy 
across divisional program lines, and the emergence of a 3yst:ems 
perspective across the Agency — with all that implies for the con- 
sideration of interaction/portfolio effects, building multi-pur-^ '^os 
(especially system building) into procurements, and de.si niing 
curements in relation to a variety of non-procurement actions 

in the long run may have greater consequence for the systeiL ti\au any 

individual procurer ,nt or set of procurements. 
The procurement as initially conceived might clearly have accomplished 
much that could be viewed as highly beneficial, and it is rot ouj: purpose to 
find fault with procurements made in this manner. Still, we belf^ve that the 
benefits likely to accrue from the redesigned procurement are substantially 
greater* Therefore, we recommend this approach to th3 Institute and the 
Council for their consideration. 
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II . UNSOLICITED PROPOSALS; A PROPOSED FRAMEWORK FOR DESIGN OF NATIONAL 



PROGRAM! EVALUATIONS 
1 . Description of the Procureme nt^ 

Among the unsolicited proposals received by NIE in mid 1976 was one 
particularly interesting description of a f rau*:'.u'ork for design of national 
program evaluations. The approach was designed to pro* '.de the kinds of pro- 
cess and impact data that could be expected to meet the information needs of 
decision makers at different levels of government, as well as program managers 
and staff. The approach permitted some degree of local variability in de- 
signs and emphases while at the same time enabl Ing data to be aggregated or 
disaggregated to meet different decision needs. What made the proposal seem 
particularly intriguing was the comprehensiveness with which program charac-- 
t eristics and implementation conditions were treated in the scheme outlined. 

The Research organization that submitted the proposal was requesting 
funds for elaboration and subsequent testing of the framework across several 
national programs varying in key program and/or implementation characteristics. 

The proposal appeared to be relevant to the program concerns of sev- 
eral NIE division,^ and was therefore reviewed by each of these in turn. In 
some cases, program personnel were not sa'''^iently interested in the proposal 
to be willing to fund it out of their own allocations. In one case, there was 
substantial interest in funding the proposal but available fundi; of that di- 
vision for the given program year had already been committed. The leadership 
of that division suggested that perhaps the needed funds might be secured from 
the budget of the Director's Office where there was available a small reserve 
fund for potentially significant funding opportunities of this kind. 

The Director's Office requested that the proposal be reviewed by sev- 
.^ral leading members of the Evaluation Research and Research Design fields. 



*While the case is presented to read as though it describes an actual NIE pro- 
curement, it is in fact hypothetical. Also, the authors imply no position on 
the substantive content of the procurement. 
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Favorable comments were received from mc^r. of these reviewers, with only 
minor points suggested here and there for rethinking. These points, along 
with some additional considerations and possible reservations, raised by a 
knowledgeable NIE staffer, were then forwarded to the proposing Research 
organization. After some discussions, a mutually agreeable Research Grant 
scope of work was drafted, calling for full development of the proposed 
scheme and field testing of its utility and effectiveness at specified sites 
of designated f edera. ly-funded national educational programs carried out under 
NIE or OE auspices. 

NIE staff had previously met with OE personnel to advise OE of the 
proposed grant, suggest and work out the details of the needed coordination 
between agencxes, and secure OE approval and cooperation in the use of 
OE-funded programs and OE-supported program sites in r.he field test. OE was 
enthusiastic about the grant since it had been supporting related work on the 
development of a set of somewhat standardized models for ESEA Title I evalua- 
tions. The work supported by OE was similar in its orientation toward making 
possible data aggregation or disaggregation as needed to meet different in- 
formation needs. But the OE-supported work was less powerful in its lack of 
attention to implementation conditions (a weakness recognized by the OE con- 
tractor and taken into account in their longer-term plans). 

2 . Impacts 

In considering what the likely impacts of this procurement mxc v ^.v.;^ , 
the following seems reasonable: 

a) The work might well produce a significant breaktbr -^u,.;.^ 

resolving some of the methodological dilemmas of the E-^'>ilua- 
tion Research function at this point in its development. 
Its positive reception from leaders of the field might be 
viewed as encouraging. However, since breakthroughs 
tend to be relatively rare and Research of this, kind in- 
volves a considerable amount '^f uncertainty, it is also 
quite possible (and perhaps even moie probable) that dif- 
ficulties encountered along the way will reveal unantv^-i- 
pated problems in the proposed framework, weakening its 
utility for widescale application. If such is the case, 
then the substantive Research output might ^ tter be 
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classified as an important (or possibly unimportant) 
part of the accumulating knowledge/technology base of 
the Evaluation Research function, and not as a key 
breakthrough in the field's methodological development. 

b) NIE*s involvement of the field in evaluation of the 
proposal is likely to be well received, at least by the 
part of the field included in the evaluation process. 
Given the NIE strategy of including the field's leader- 
ship in this process, the payoff in generating positive 
field affect is likely to be substantial (unless of 
course the field is undergoing a period of change or is 
divided into different "schools of thought" and NIE's 
selection of advisers takes into account only one of 
several groups vying for influence on the direction of 
the field's development). 

c) NIE-OE coordination of efforts is likely to strengthen 
the links between these two key education agencies. 

3. Evaluation of the Procurement and its Impacts, Actual and .Potential 

In evaluating the procurement. Agency person el must weigh actual 
(or likely) impacts of a procurement as initially formulated with possible 
desired impacts. The question to consider here is: How might such a 
procurement be designed to better achieve Agency purposes, or to achieve 
a broader ranga of Agf^^ncy purposes? 

At the very lease this question requires insuring some clarity on 
Agency purposes: What in fact are the Agency purposes that are viewed as 
relevant to this procurement? Might a somewhat different view of the 
Agency rol'e and its requirements for procurement policies produce a 
different or an enlarged picture of the relevant purposes? 

At the very least, NIE's role is conceived as that of a funding 
^ency which can facilitate quality work in the field through choices as 
to which work it fund's! Therefore, the unsolicited proposal would likely 
have been evaluatec' in accord with some of the criteria we suggested in 
our discussion of the Research function: Does the proposal fall within a 
Research area with which we have decided to work? If it does, can the 
proposal be judged to demonstrate the level of state-of-the-art creativity 



and Excellence that calls for support. (If it falls outside a Research 
area that we have decided to work with, but is of such exc^illent quality 
that NIE should in some way or other be part of its sponsorship, how 
might NIE work with other agencies to insure and to coordinate its sup- 
port?) Does NIE have the kinds of people in-house who can work with 
the proposing Researcher (s)? Which Researcher (s) and which organiza- 
tion (s) will be supported if the proposal is funded — a "star" of the 
field? an exciting new talent? an existing center of excellence with 
the needed minimum critical mass of relevant talent? or, an institu- 
tional setting lacking in the organizational resources needed to ade- 
quately carry out the proposal or build long-term system capacity? 

Once we start considering this latter group of questions, we have 
moved considerably beyond a conception of NIE's role as simply a funding 
agency seeking to identify individual proposals to fund. Instead, NIE's 
role is viewed in terms of broader, more system-oriented responsibilities. 

One of our basic coitentions in our discussion of the Research 
function was that the latent, long-term, system-building potential _f any 
Research procurement was likely to be tar more consequential than the 
manifest substantive purpose of the procurement. If this argument is 
accepted, and if we define the relevant Research area as Evaluation 
Research methodology, then sv:-.veral aspects of the Evaluation ^ese'trch 
function as it exists in the education sector today become relevant to 
evaluating the proposed approach to making this procur.-^ment : 

a) The knowledge/ technology base of the field has been 
undergclng extensive development in recent years. 

b) Ihere is a clearly visible Evaluation Research cDmmu- 
nity, with its own leadership, its own cliannels of 
communication, its developing standards of quality 
work, etc. 

c) Though the numberii of Evaluation Researchers and in- 
stitutional bas^s for Evaluation Research are large, 
the ani.oi'rit of first-rate talent is somewhat limited anc 
li. dj.^^.. jubuted unc=»,venly across the various types of per- 
former organizatioTifi in the field. 

d) There is a huge j'eciiixal investment in Evaluation Research, 
especially if one aggregates Evaluation Research 
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expenditures across agencies* 
e) There is substantial foundation interest (the Russell 
Sage Foundation) in the development of Evaluation Re- 
search methodology. 
Therefore, it seems reasonable to consider how much greater the po- 
tential impact of this unsolicited proposal might be if it was used as a 
base on which to build a broader scale, longer-term, staged, multi-purpose 
program, coordinated across governmental agencies and internally within 
NIE across divisions and program boundary lines. 

4. Possible Alternatives 

A. Work with the Proposing Researcher in an Interactive/ Collaborative Mode 
If NIE had Stirong in-house staff with the needed methodological skills 

(and there would seem to be some staffers with these talents at present) , 
the Agency could work with the proposing Researcher to strengthen the pro- 
posal, relate it to other work going on in the field and other relevant in- 
itiatives funded by other agencies (e.g., OE) . In-house personnel could 
possibly: facilitate information exchanges bs.:itween the proposing Researcher 
and others in the field; develop meeded syntheses and critical reviews 
mapping the area and uhe state of the xt, useful for stimulating field com- 
munication about the proposed framework (and che ongoing work to elaborate 
and test it) ; make data available for use in elaborating or testing the 
framework's utility prior to field work; etc. 

B. Using the Proposal as a Basis for Developing Close Working Relation- 

with the Stronp Talent in the Field 
NIE might consider convening a continuing seminar or panel to discuss 
the implications of the proposed (and subsequently the ongoing) work, to 
assess its quality, and to consider what it does or does not suggest about 
directions for the methodological development of the field as well as the 
work's practical applications in national program evaluations. This kind 
of ongoing seminar might become a wedge for desigi of alternative ap- 
proaches ( or components of a single unified approach) , each carried out 
by different participants in the seminar, alone or working together 
collaboratively. 




C> Cross-Agency/Funder Coordination 

NIE might play a leadership role in contacting other i^rn^lcs jr 
funders (e.g., the Russell Sage Foundation) to stimulate th^ interest 
in this proposal as well as the possibilities of identifyi^^ >/nergistic 
add-ons or points of relevance to work sponsored by these : r. :>: r t-jiaders. 
Possibilities for pooling of resources might be a particul.ir print of 
focus in these interactions. 

D. Internal NIE 

All NIE project officers might be asked to consider tr,r ir orogr^iz^ 
evaluation needs and how these might relate to the proposed r r .i.-:cvc . 
Such discussions — in-house, with the proposing Researcher, : m i -ith the 
long-term advisory panel of leaders of the Evaluation Re^*.i^^' rleld — 
would seem to have the potential to stimulate cross-prcgr.r:.'. :™ur.ication 
and synergy in the development and interrelation of Eval!:.i i »eo. :h 

designs for NIE programs. 

5. Summary Description of the Alternative Procurement ir.i r 1 i r cd Ac t i/itie ' 

A. Staging and Field/Agency Relations 

The procurement and r^^lated activities could t:io i,;;r ; : aving 
the following possible elements: 

1) collaborative/interactive NIE~Research^-r ^ - 1 : pi:ent of 
the proposal in its final form; 

2) NIE staff development of state of t:vj ; r ' . • • -s:es and 
critiques that relate the proposed fr.ir-. - t > the ex- 
isting technology base of the tie±^: 

3) convening of a long-term, on-goin^ .i-'i* .-iters of 
the Evaluation Research nnd Resear ... [n . :n:L:init ies 
to assess the validity^ significance , . ; r . L icn t : .:ns 

of the ongoing woric as well as to • - - ir^- "he devel- 
opment of an "invisible coil ge" tnec ha- !,'>:-: 



*We wish to remind tihe reader of the hypothetical ■:hcir:i:zr-r the 
process described and our neutral position as to the ^jb:.r i r.tent 
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4) design of alternative models, or system components of 
a unified model, with different members of the seminar 
group working on pieces of this, individually or col- 
laboratively; 

5) testing of the initially proposed framework (as elab- 
orated) , and possibly alternative models, through 
reanalysis of available Evaluation Research data sets; 

6) field testing of these frameworks/models in a small 
sample of selected program sites; 

7) large-scale field testing across varying programs 
nationwide; 

8) consideration of utility of developed models at interim 
points in the testing procedure; 

9) convening conferences of Evaluation Researchers and pro- 
gram personnel from varying national educational programs 
to consider the implications of this work. 

Several of these stages might be clustered together, but what is impor- 
ant is' that each stage suggests critical questions that will need answers 
before it will be obvious as to whether it is justifiable to proceed. 

B. Multi-Purposes 

The manifest purpose of this procurement — i.e., to elaborate and test 
the proposed framework — appears to be of some consequence in and of itself. 
But in addition, other latent purposes could be achieved as well if the pro- 
curement was considered in system-oriented terms. We have noted several 
ways in which system-building purposes could be achieved (e.g., facilitating 
communication among, and providing additional support for the work of, the 
sirong Research talent and Research organizations in the field) . In addition 
••Positive environmental impact might accrue from involvement of the program 
personnel in the assessment of the new framework and its implications after 
the field test results are available for consideration. 

6> Likely Impacts 

a) nib's close working relationship to the field in evaluating and 
.hen expanding the scope of this proposal is likely to strengthen 
nib's position vis a vis the field and enhance the Agency's image 
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as an integral and key part of the field facilitating its 
development. 

b) As we have noted, several system-building impacts seem likely: 
developing communication linkages among the strong talent in the 
Evaluation Research and Research Design fields (e.g., "invisible 
college" mechanisms); providing additional support for and facil- 
itating the quality work of strong Research organizations working 
in relevant Research areas; and affecting the long-term Research 
agendas of these strong organizations and increasing the potential 
for synergy across the work of these various organizations (and 
betwG a their agendas and NIE's), 

c) NIE's position as the lead agency for Research and Development in • 
education is likely to be strengthened by cross-agency coordination 
of the kind s^pggested here. 

d) Increased communication among NIE program officers (considering the 
implications of this proposal to their programs) is likely to 
strengthen the development of stronger program evaluations that con- 
sider interaction effects across programs as well as Impacts at- 
tributable to discrete programs. 

Our conclusion here is similar to our statement in the previous scenario. 
The procurement as initially conceived might clearly have accomplished much 
that could be viewed as highly beneficial. And, clearly, much that we have 
suggested is similar to courses of action frequently followed by the Institute. 
What may be different here, and what may explain any additional benefits 
likely to accrue from the expanded description of the rethought procurement 
and non-procurement actions, is the emphasis on simultaneity and interaction 
among these occasionally considered options. We therefore recommend this 
kind of thinking to the Institute and the Couacil for consideration. 



235 



